Journal of Computer Science and Technology ›› 2022, Vol. 37 ›› Issue (1): 83-105.doi: 10.1007/s11390-021-1669-1

Special Issue: Software Systems

• Special Section on Software Systems 2021 • Previous Articles     Next Articles

Simulation Might Change Your Results: A Comparison of Context-Aware System Input Validation in Simulated and Physical Environments

Jin-Chi Chen (陈金池), Yi Qin* (秦逸), Member, CCF, ACM, Hui-Yan Wang (王慧妍), Member, CCF, and Chang Xu* (许畅), Senior Member, CCF, IEEE, Member, ACM        

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China
  • Received:2021-06-01 Revised:2021-11-03 Accepted:2021-12-23 Online:2022-01-28 Published:2022-01-28
  • Contact: Yi Qin,Chang Xu E-mail:{yiqincs, changxu}@nju.edu.cn
  • About author:Yi Qin is currently an assistant researcher with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology at Nanjing University, Nanjing. He received his Ph.D. degree in computer science and technology from Nanjing University, Nanjing, in 2018. His research interests include self-adaptive software systems, and software testing and analysis.
    Chang Xu is currently a full professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology at Nanjing University, Nanjing. He received his Ph.D. degree in computer science and engineering from The Hong Kong University of Science and Technology, Hong Kong, in 2008. His research interests include big data software engineering, intelligent software testing and analysis, and adaptive and autonomous software systems. He co-authored over 160 peer-reviewed papers and served in technical program committees of various international software engineering conferences. He is a member of ACM, and a senior member of CCF and IEEE.
  • Supported by:
    This work was partially supported by the National Natural Science Foundation of China under Grant No.61932021, the Leading-Edge Technology Program of Jiangsu Natural Science Foundation of China under Grant No.BK20202001, the National Natural Science Foundation of China under Grant No.61902173, and the Natural Science Foundation of Jiangsu Province of China under Grant No.BK20190299.

Context-aware systems (a.k.a. CASs) integrate cyber and physical space to provide adaptive functionalities in response to changes in context. Building context-aware systems is challenging due to the uncertain running environment. Therefore, many input validation approaches have been proposed to protect context-aware systems from uncertainty and keep them executing safely. However, in contrast to context-aware systems' prevailing in physical environments, most of those academic solutions (83%) are purely evaluated in simulated environments. In this article, we study whether this evaluation setting could lead to biased conclusions. We build a testing platform, RM-Testing, based on DJI RoboMaster robot car, to conduct the physical-environment based experiments. We select three up-to-date input validation approaches, and compare their performance in the simulated environment and in the physical environment. The experimental results show that all three approaches' performance in simulated environments (improving task success rate by 82% compared with the system without the support of input validation) does differ from their performance in a physical environment (improving the task success rate by 50%). We also recognize three factors (scenario setting, physical platform and environmental model) that affect the performance of input validation approaches, based on an execution model of the context-aware system.

Key words: context-aware system; input validation; self-driving car; testing infrastructure;

[1] Lü J, Ma X, Huang Y, Cao C, Xu F. Internetware: A shift of software paradigm. In Proc. the 1st Asia-Pacific Symposium on Internetware, October 2009, Article No.7. DOI: 10.1145/1640206.1640213.
[2] Sama M, Elbaum S, Raimondi F, Rosenblum D S, Wang Z. Context-aware adaptive applications: Fault patterns and their automated identification. IEEE Transactions on Software Engineering, 2010, 36(5): 644-661. DOI: 10.1109/TSE.2010.35.
[3] Qin Y, Xu C, Chen Z, Lü J. Software testing for cyber-physical systems suffering uncertainty. SCIENTIA SINICA Informationis, 2019, 49(11): 1428-1450. DOI: 10.1360/N112018-00305. (in Chinese)
[4] Sama M, Rosenblum D S, Wang Z, Elbaum S. Model-based fault detection in context-aware adaptive applications. In Proc. the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, November 2008, pp.261-271. DOI: 10.1145/1453101.1453136.
[5] Xu C, Cheung S C, Ma X, Cao C, Lu J. Adam: Identifying defects in context-aware adaptation. Journal of Systems and Software, 2012, 85(12): 2812-2828. DOI: 10.1016/j.jss.2012.04.078.
[6] Xu C, Cheung S C, Chan W K, Ye C. Partial constraint checking for context consistency in pervasive computing. ACM Transactions on Software Engineering and Methodology, 2010, 19(3): Article No.9. DOI: 10.1145/1656250.1656253.
[7] Qin Y, Xie T, Xu C, Astorga A, Lu J. CoMID: Context-based multiinvariant detection for monitoring cyber-physical software. IEEE Transactions on Reliability, 2019, 69(1): 106-123. DOI: 10.1109/TR.2019.2933324.
[8] Wang H, Xu J, Xu C, Ma X, Lu J. DISSECTOR: Input validation for deep learning applications by crossing-layer dissection. In Proc. the 42nd ACM/IEEE International Conference on Software Engineering, Oct. 2020, pp.727-738. DOI: 10.1145/3377811.3380379.
[9] Ramirez A J, Jensen A C, Cheng B H, Knoester D B. Automatically exploring how uncertainty impacts behavior of dynamically adaptive systems. In Proc. the 26th IEEE/ACM International Conference on Automated Software Engineering, November 2011, pp.568-571. DOI: 10.1109/ASE.2011.6100127.
[10] Aliabadi M R, Kamath A A, Gascon-Samson J, Pattabiraman K. ARTINALI: Dynamic invariant detection for cyber-physical system security. In Proc. the 11th Joint Meeting on Foundations of Software Engineering, Sept. 2017, pp.349-361. DOI: 10.1145/3106237.3106282.
[11] Wang Z, Elbaum S, Rosenblum D S. Automated generation of context-aware tests. In Proc. the 29th International Conference on Software Engineering, May 2007, pp.406-415. DOI: 10.1109/ICSE.2007.18.
[12] Nentwich C, Capra L, Emmerich W, Finkelsteiin A. xlinkit: A consistency checking and smart link generation service. ACM Transactions on Internet Technology, 2002, 2(2): 151-185. DOI: 10.1145/514183.514186.
[13] Egyed A. Fixing inconsistencies in UML design models. In Proc. the 29th International Conference on Software Engineering, May 2007, pp.292-301. DOI: 10.1109/ICSE.2007.38.
[14] Chu X, Morcos J, Ilyas I F, Ouzzani M, Papotti P, Tang N, Ye Y. KATARA: A data cleaning system powered by knowledge bases and crowdsourcing. In Proc. the 2015 ACM SIGMOD International Conference on Management of Data, May 31-June 4, 2015, pp.1247-1261. DOI: 10.1145/2723372.2749431.
[15] Pei K, Cao Y, Yang J, Jana S. DeepXplore: Automated whitebox testing of deep learning systems. In Proc. the 26th Symposium on Operating Systems Principles, October 2017, pp.1-18. DOI: 10.1145/3132747.3132785.
[16] Tian Y, Pei K, Jana S, Ray B. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proc. the 40th International Conference on Software Engineering, May 27--June 3, 2018, pp.303-314. DOI: 10.1145/3180155.3180220.
[17] Luo C, Goncalves J, Velloso E, Kostakos V. A survey of context simulation for testing mobile context-aware applications. ACM Computing Surveys, 2020, 53(1): Article No.21. DOI: 10.1145/3372788.
[18] Chen J, Qin Y, Wang H, Xu C. Simulated or physical? An empirical study on input validation for context-aware systems in different environments. In Proc. the 12th Asia-Pacific Symposium on Internetware, Nov. 2021, pp.146-155. DOI: 10.1145/3457913.3457919.
[19] Paden B, Čáp M, Yong S Z, Yershov D, Frazzoli E. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on Intelligent Vehicles, 2016, 1(1): 33-55. DOI: 10.1109/TIV.2016.2578706.
[20] Badue C, Guidolini R, Carneiro R V et al. Self-driving cars: A survey. Expert Systems with Applications, 2021, 165: Article No.113816. DOI: 10.1016/j.eswa.2020.113816.
[21] Ma N, Gao Y, Li J, Li D. Interactive cognition in self-driving. SCIENTIA SINICA Informationis, 2018, 48(8): 1083-1096. DOI: 10.1360/N112018-00028. (in Chinese)
[22] Xi W, Xu C, Yang W, Hong X. How context inconsistency and its resolution impact context-aware applications. Journal of Frontiers of Computer Science and Technology, 2014, 8(4): 427-437. DOI: 10.3778/j.issn.1673-9418.1311013. (in Chinese)
[23] Esfahani N, Malek S. Uncertainty in self-adaptive software systems. In Proc. the International Seminar on Software Engineering for Self-Adaptive Systems, Oct. 2013, pp.214-238. DOI: 10.1007/978-3-642-35813-5-9.
[24] Fredericks E M, DeVries B, Cheng B H C. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty. In Proc. the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, June 2014, pp.17-26. DOI: 10.1145/2593929.2593937.
[25] Jiang H, Elbaum S, Detweiler C. Reducing failure rates of robotic systems though inferred invariants monitoring. In Proc. the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, November 2013, pp.1899-1906. DOI: 10.1109/IROS.2013.6696608.
[26] Al-Garadi M A, Mohamed A, Al-Ali A K, Du X, Ali I, Guizani M. A survey of machine and deep learning methods for Internet of Things (IoT) security. IEEE Communications Surveys {\rm \&] Tutorials, 2020, 22(3): 1646-1685. DOI: 10.1109/COMST.2020.2988293.
[27] Li Z, Ma X, Xu C, Cao C, Xu J, Lü J. Boosting operational DNN testing efficiency through conditioning. In Proc. the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, August 2019, pp.499-509. DOI: 10.1145/3338906.3338930.
[28] Li Z, Ma X, Xu C, Xu J, Cao C, Lü J. Operational calibration: Debugging confidence errors for DNNs in the field. In Proc. the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, November 2020, pp.901-913. DOI: 10.1145/3368089.3409696.
[29] Nentwich C, Emmerich W, Finkelsteiin A, Ellmer E. Flexible consistency checking. ACM Transactions on Software Engineering and Methodology, 2003, 12(1): 28-63. DOI: 10.1145/839268.839271.
[30] Gehrke J, Madden S. Query processing in sensor networks. IEEE Pervasive Computing, 2004, 3(1): 46-55. DOI: 10.1109/MPRV.2004.1269131.
[31] Tong Y, Qin Y, Jiang Y, Xu C, Cao C, Ma X. Timely and accurate detection of model deviation in self-adaptive software-intensive systems. In Proc. the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, August 2021, pp.168-180. DOI: 10.1145/3468264.3468548.
[32] Ernst M D, Perkins J H, Guo P J, McCamant S, Pacheco C, Tschantz M S, Xiao C. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, 2007, 69(1/2/3): 35-45. DOI: /10.1016/j.scico.2007.01.015.
[33] Jiang H, Elbaum S, Detweiler C. Inferring and monitoring invariants in robotic systems. Autonomous Robots, 2017, 41(4): 1027-1046. DOI: 10.1007/s10514-016-9576-y.
[34] Bai X, Yang M, Shi B, Liao M. Deep learning for scene text detection and recognition. SCIENTIA SINICA Informationis, 2018, 48(5): 531-544. DOI: 10.1360/N112018-00003. (in Chinese)
[35] Matalonga S, Travassos G H. Testing context-aware software systems: Unchain the context, set it free! In Proc. the 31st Brazilian Symposium on Software Engineering, September 2017, pp.250-254. DOI: 10.1145/3131151.3131190.
[36] Dourish P. What we talk about when we talk about context. Personal and Ubiquitous Computing, 2004, 8(1): 19-30. DOI: 10.1007/s00779-003-0253-8.
[37] Ramirez A J, Jensen A C, Cheng B H C. A taxonomy of uncertainty for dynamically adaptive systems. In Proc. the 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, June 2012, pp.99-108. DOI: 10.1109/SEAMS.2012.6224396.
[38] Yang W, Xu C, Liu Y, Cao C, Ma X, Lu J. Verifying self-adaptive applications suffering uncertainty. In Proc. the 29th ACM/IEEE International Conference on Automated Software Engineering, September 2014, pp.199-210. DOI: 10.1145/2642937.2642999.
[39] Wohlin C, Runeson P, Höst M, Ohlsson M C, Regnell B, Wessl\'{e}n A. Experimentation in Software Engineering. Springer, 2000. DOI: 10.1007/978-3-642-29044-2.
[40] Zhang L, Xu C, Ma X, Gu T, Hong X, Cao C, Lu J. Resynchronizing model-based self-adaptive systems with environments. In Proc. the 19th Asia-Pacific Software Engineering Conference, December 2012, pp.184-193. DOI: 10.1109/APSEC.2012.62.
[41] Dey A K, Abowd G D, Salber D. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-Computer Interaction, 2001, 16(2): 97-166. DOI: 10.1207/S15327051HCI16234-02.
[42] Magnusson S. Similarities and differences in psychophysiological reactions between simulated and real air-to-ground missions. The International Journal of Aviation Psychology, 2002, 12(1): 49-61. DOI: 10.1207/S15327108IJAP1201-5.
[43] Bishop I D, Rohrmann B. Subjective responses to simulated and real environments: A comparison. Landscape and Urban Planning, 2003, 65(4): 261-277. DOI: 10.1016/S0169-2046(03)00070-7.
[44] Qin Y, Xu C, Yu P, Lu J. SIT: Sampling-based interactive testing for self-adaptive apps. Journal of Systems and Software, 2016, 120: 70-88. DOI: 10.1016/j.jss.2016.07.002.
[45] Weyns D, Schmerl B, Grassi V, Malek S, Mirandola R, Prehofer C, Wuttke J, Andersson J, Giese H, Göschka K M. On patterns for decentralized control in self-adaptive systems. Software Engineering for Self-Adaptive Systems II, 2013, pp.76-107. DOI: 10.1007/978-3-642-35813-5-4.
[46] Kephart J O, Chess D M. The vision of autonomic computing. Computer, 2003, 36(1): 41-50. DOI: 10.1109/MC.2003.1160055.
No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] . Online First Under Construction [J]. Journal of Computer Science and Technology, 0, (): 1 .
[2] Zhi-Neng Chen, Chong-Wah Ngo, Wei Zhang, Juan Cao, Yu-Gang Jiang. Name-Face Association in Web Videos: A Large-Scale Dataset, Baselines, and Open Issues[J]. , 2014, 29(5): 785 -798 .
[3] Fei Xia, De-Jun Jiang, Jin Xiong, Ning-Hui Sun. A Survey of Phase Change Memory Systems[J]. , 2015, 30(1): 121 -144 .
[4] André Brinkmann, Kathryn Mohror, Weikuan Yu, Philip Carns, Toni Cortes, Scott A. Klasky, Alberto Miranda, Franz-Josef Pfreundt, Robert B. Ross, Marc-André Vef. Ad Hoc File Systems for High-Performance Computing[J]. Journal of Computer Science and Technology, 2020, 35(1): 4 -26 .
[5] Yu-Tong Lu, Peng Cheng, Zhi-Guang Chen. Design and Implementation of the Tianhe-2 Data Storage and Management System[J]. Journal of Computer Science and Technology, 2020, 35(1): 27 -46 .
[6] Reza Jafari Ziarani, Reza Ravanmehr. Serendipity in Recommender Systems: A Systematic Literature Review[J]. Journal of Computer Science and Technology, 2021, 36(2): 375 -396 .
[7] Bo-Han Li, Yi Liu, An-Man Zhang, Wen-Huan Wang, Shuo Wan. A Survey on Blocking Technology of Entity Resolution[J]. Journal of Computer Science and Technology, 2020, 35(4): 769 -793 .
[8] Lie-Huang Zhu, Bao-Kun Zheng, Meng Shen, Feng Gao, Hong-Yu Li, Ke-Xin Shi. Data Security and Privacy in Bitcoin System: A Survey[J]. Journal of Computer Science and Technology, 2020, 35(4): 843 -862 .
[9] Dun Liang, Yuan-Chen Guo, Shao-Kui Zhang, Tai-Jiang Mu, Xiaolei Huang. Lane Detection: A Survey with New Results[J]. Journal of Computer Science and Technology, 2020, 35(3): 493 -505 .
[10] Lan Huang, Da-Lin Li, Kang-Ping Wang, Teng Gao, Adriano Tavares. A Survey on Performance Optimization of High-Level Synthesis Tools[J]. Journal of Computer Science and Technology, 2020, 35(3): 697 -720 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

Home
Editorial Board
Author Guidelines
Subscription
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved