计算机科学技术学报 ›› 2022,Vol. 37 ›› Issue (1): 83-105.doi: 10.1007/s11390-021-1669-1

所属专题： Software Systems

• • 上一篇    下一篇

上下文感知系统输入验证方法在模拟环境与物理环境中的对比

  

  • 收稿日期:2021-06-01 修回日期:2021-11-03 接受日期:2021-12-23 出版日期:2022-01-28 发布日期:2022-01-28

Simulation Might Change Your Results: A Comparison of Context-Aware System Input Validation in Simulated and Physical Environments

Jin-Chi Chen (陈金池), Yi Qin* (秦逸), Member, CCF, ACM, Hui-Yan Wang (王慧妍), Member, CCF, and Chang Xu* (许畅), Senior Member, CCF, IEEE, Member, ACM        

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China
  • Received:2021-06-01 Revised:2021-11-03 Accepted:2021-12-23 Online:2022-01-28 Published:2022-01-28
  • Contact: Yi Qin，Chang Xu E-mail:{yiqincs, changxu}@nju.edu.cn
  • About author:Yi Qin is currently an assistant researcher with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology at Nanjing University, Nanjing. He received his Ph.D. degree in computer science and technology from Nanjing University, Nanjing, in 2018. His research interests include self-adaptive software systems, and software testing and analysis.
    Chang Xu is currently a full professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology at Nanjing University, Nanjing. He received his Ph.D. degree in computer science and engineering from The Hong Kong University of Science and Technology, Hong Kong, in 2008. His research interests include big data software engineering, intelligent software testing and analysis, and adaptive and autonomous software systems. He co-authored over 160 peer-reviewed papers and served in technical program committees of various international software engineering conferences. He is a member of ACM, and a senior member of CCF and IEEE.
  • Supported by:
    This work was partially supported by the National Natural Science Foundation of China under Grant No.61932021, the Leading-Edge Technology Program of Jiangsu Natural Science Foundation of China under Grant No.BK20202001, the National Natural Science Foundation of China under Grant No.61902173, and the Natural Science Foundation of Jiangsu Province of China under Grant No.BK20190299.

PDF (PC)

21

补充材料

1

1、研究背景
上下文感知系统（Context-aware systems, CAS）融合了软件组件和物理环境。它不断地感知物理环境环境的上下文变化，基于预定义的逻辑做决策，并执行相应的动作以应对环境变化。由于传感器噪声、机械偏差等物理环境中天然存在的问题，CAS通常面临与环境交互过程中的不确定性问题。因此，许多现有的研究工作采用对CAS的输入数据做验证的方法（即输入验证方法）来缓解这一问题。
与传统软件不同，CAS通常在物理环境中运行。然而，根据我们的一项调研，相关文献中只有17%的工作在物理环境中评估了他们的方法，其余工作中提出的方法只在模拟环境中（或只使用预先收集的执行轨迹）评估。根据近期一篇文献综述中的讨论，这可能是因为在物理环境中测试CAS需要付出更加高昂的时间和金钱成本。
2、目的
在这项工作中，我们尝试回答以下两个问题：
(1) CAS输入验证方法在物理环境中的表现与其在模拟环境中的表现是否有差异？
(2) 若(1)的结论为“有差异”，那么物理环境和模拟环境的哪些不同点会影响这一差异？
3、方法
我们分别在物理环境（基于大疆RoboMaster机器人小车）和模拟环境（基于Unity引擎）中搭建了无人驾驶汽车测试演示场景，并接入已有工作中提出的三种输入验证方法：约束检测（Constraint checking），不变式检测（Invariant checking）和深度学习模型输入裁剪（Deep learning model input pruning）。我们分别在物理环境和模拟环境中开展相同配置的实验，比较这两种环境中CAS输入验证方法对小车完成任务比例的提高的差异。
4、结果
总体上，物理环境中三种CAS输入验证方法的开启能使小车完成任务的比例提高82%，而在模拟环境中提高的比例为50%。这一结果表明CAS输入验证方法在物理环境和模拟环境中的表现存在差异。细粒度的分析还表明场景设置、物理平台和环境模型等三个因素会对CAS输入验证方法在两种环境中的有效性造成不同程度的影响。
5、结论
CAS输入验证方法在物理环境和模拟环境中的表现可能存在差异，且受场景设置、物理平台和环境模型等因素的影响。因此，面向CAS的方法应该在物理环境中评估，以避免模拟环境带来的偏差。

关键词: 上下文感知系统, 自动驾驶汽车, 输入验证, 软件测试

Abstract: Context-aware systems (a.k.a. CASs) integrate cyber and physical space to provide adaptive functionalities in response to changes in context. Building context-aware systems is challenging due to the uncertain running environment. Therefore, many input validation approaches have been proposed to protect context-aware systems from uncertainty and keep them executing safely. However, in contrast to context-aware systems' prevailing in physical environments, most of those academic solutions (83%) are purely evaluated in simulated environments. In this article, we study whether this evaluation setting could lead to biased conclusions. We build a testing platform, RM-Testing, based on DJI RoboMaster robot car, to conduct the physical-environment based experiments. We select three up-to-date input validation approaches, and compare their performance in the simulated environment and in the physical environment. The experimental results show that all three approaches' performance in simulated environments (improving task success rate by 82% compared with the system without the support of input validation) does differ from their performance in a physical environment (improving the task success rate by 50%). We also recognize three factors (scenario setting, physical platform and environmental model) that affect the performance of input validation approaches, based on an execution model of the context-aware system.

Key words: context-aware system, input validation, self-driving car, testing infrastructure

[1] Lü J, Ma X, Huang Y, Cao C, Xu F. Internetware: A shift of software paradigm. In Proc. the 1st Asia-Pacific Symposium on Internetware, October 2009, Article No.7. DOI: 10.1145/1640206.1640213.
[2] Sama M, Elbaum S, Raimondi F, Rosenblum D S, Wang Z. Context-aware adaptive applications: Fault patterns and their automated identification. IEEE Transactions on Software Engineering, 2010, 36(5): 644-661. DOI: 10.1109/TSE.2010.35.
[3] Qin Y, Xu C, Chen Z, Lü J. Software testing for cyber-physical systems suffering uncertainty. SCIENTIA SINICA Informationis, 2019, 49(11): 1428-1450. DOI: 10.1360/N112018-00305. (in Chinese)
[4] Sama M, Rosenblum D S, Wang Z, Elbaum S. Model-based fault detection in context-aware adaptive applications. In Proc. the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, November 2008, pp.261-271. DOI: 10.1145/1453101.1453136.
[5] Xu C, Cheung S C, Ma X, Cao C, Lu J. Adam: Identifying defects in context-aware adaptation. Journal of Systems and Software, 2012, 85(12): 2812-2828. DOI: 10.1016/j.jss.2012.04.078.
[6] Xu C, Cheung S C, Chan W K, Ye C. Partial constraint checking for context consistency in pervasive computing. ACM Transactions on Software Engineering and Methodology, 2010, 19(3): Article No.9. DOI: 10.1145/1656250.1656253.
[7] Qin Y, Xie T, Xu C, Astorga A, Lu J. CoMID: Context-based multiinvariant detection for monitoring cyber-physical software. IEEE Transactions on Reliability, 2019, 69(1): 106-123. DOI: 10.1109/TR.2019.2933324.
[8] Wang H, Xu J, Xu C, Ma X, Lu J. DISSECTOR: Input validation for deep learning applications by crossing-layer dissection. In Proc. the 42nd ACM/IEEE International Conference on Software Engineering, Oct. 2020, pp.727-738. DOI: 10.1145/3377811.3380379.
[9] Ramirez A J, Jensen A C, Cheng B H, Knoester D B. Automatically exploring how uncertainty impacts behavior of dynamically adaptive systems. In Proc. the 26th IEEE/ACM International Conference on Automated Software Engineering, November 2011, pp.568-571. DOI: 10.1109/ASE.2011.6100127.
[10] Aliabadi M R, Kamath A A, Gascon-Samson J, Pattabiraman K. ARTINALI: Dynamic invariant detection for cyber-physical system security. In Proc. the 11th Joint Meeting on Foundations of Software Engineering, Sept. 2017, pp.349-361. DOI: 10.1145/3106237.3106282.
[11] Wang Z, Elbaum S, Rosenblum D S. Automated generation of context-aware tests. In Proc. the 29th International Conference on Software Engineering, May 2007, pp.406-415. DOI: 10.1109/ICSE.2007.18.
[12] Nentwich C, Capra L, Emmerich W, Finkelsteiin A. xlinkit: A consistency checking and smart link generation service. ACM Transactions on Internet Technology, 2002, 2(2): 151-185. DOI: 10.1145/514183.514186.
[13] Egyed A. Fixing inconsistencies in UML design models. In Proc. the 29th International Conference on Software Engineering, May 2007, pp.292-301. DOI: 10.1109/ICSE.2007.38.
[14] Chu X, Morcos J, Ilyas I F, Ouzzani M, Papotti P, Tang N, Ye Y. KATARA: A data cleaning system powered by knowledge bases and crowdsourcing. In Proc. the 2015 ACM SIGMOD International Conference on Management of Data, May 31-June 4, 2015, pp.1247-1261. DOI: 10.1145/2723372.2749431.
[15] Pei K, Cao Y, Yang J, Jana S. DeepXplore: Automated whitebox testing of deep learning systems. In Proc. the 26th Symposium on Operating Systems Principles, October 2017, pp.1-18. DOI: 10.1145/3132747.3132785.
[16] Tian Y, Pei K, Jana S, Ray B. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proc. the 40th International Conference on Software Engineering, May 27--June 3, 2018, pp.303-314. DOI: 10.1145/3180155.3180220.
[17] Luo C, Goncalves J, Velloso E, Kostakos V. A survey of context simulation for testing mobile context-aware applications. ACM Computing Surveys, 2020, 53(1): Article No.21. DOI: 10.1145/3372788.
[18] Chen J, Qin Y, Wang H, Xu C. Simulated or physical? An empirical study on input validation for context-aware systems in different environments. In Proc. the 12th Asia-Pacific Symposium on Internetware, Nov. 2021, pp.146-155. DOI: 10.1145/3457913.3457919.
[19] Paden B, Čáp M, Yong S Z, Yershov D, Frazzoli E. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on Intelligent Vehicles, 2016, 1(1): 33-55. DOI: 10.1109/TIV.2016.2578706.
[20] Badue C, Guidolini R, Carneiro R V et al. Self-driving cars: A survey. Expert Systems with Applications, 2021, 165: Article No.113816. DOI: 10.1016/j.eswa.2020.113816.
[21] Ma N, Gao Y, Li J, Li D. Interactive cognition in self-driving. SCIENTIA SINICA Informationis, 2018, 48(8): 1083-1096. DOI: 10.1360/N112018-00028. (in Chinese)
[22] Xi W, Xu C, Yang W, Hong X. How context inconsistency and its resolution impact context-aware applications. Journal of Frontiers of Computer Science and Technology, 2014, 8(4): 427-437. DOI: 10.3778/j.issn.1673-9418.1311013. (in Chinese)
[23] Esfahani N, Malek S. Uncertainty in self-adaptive software systems. In Proc. the International Seminar on Software Engineering for Self-Adaptive Systems, Oct. 2013, pp.214-238. DOI: 10.1007/978-3-642-35813-5-9.
[24] Fredericks E M, DeVries B, Cheng B H C. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty. In Proc. the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, June 2014, pp.17-26. DOI: 10.1145/2593929.2593937.
[25] Jiang H, Elbaum S, Detweiler C. Reducing failure rates of robotic systems though inferred invariants monitoring. In Proc. the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, November 2013, pp.1899-1906. DOI: 10.1109/IROS.2013.6696608.
[26] Al-Garadi M A, Mohamed A, Al-Ali A K, Du X, Ali I, Guizani M. A survey of machine and deep learning methods for Internet of Things (IoT) security. IEEE Communications Surveys {\rm \&] Tutorials, 2020, 22(3): 1646-1685. DOI: 10.1109/COMST.2020.2988293.
[27] Li Z, Ma X, Xu C, Cao C, Xu J, Lü J. Boosting operational DNN testing efficiency through conditioning. In Proc. the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, August 2019, pp.499-509. DOI: 10.1145/3338906.3338930.
[28] Li Z, Ma X, Xu C, Xu J, Cao C, Lü J. Operational calibration: Debugging confidence errors for DNNs in the field. In Proc. the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, November 2020, pp.901-913. DOI: 10.1145/3368089.3409696.
[29] Nentwich C, Emmerich W, Finkelsteiin A, Ellmer E. Flexible consistency checking. ACM Transactions on Software Engineering and Methodology, 2003, 12(1): 28-63. DOI: 10.1145/839268.839271.
[30] Gehrke J, Madden S. Query processing in sensor networks. IEEE Pervasive Computing, 2004, 3(1): 46-55. DOI: 10.1109/MPRV.2004.1269131.
[31] Tong Y, Qin Y, Jiang Y, Xu C, Cao C, Ma X. Timely and accurate detection of model deviation in self-adaptive software-intensive systems. In Proc. the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, August 2021, pp.168-180. DOI: 10.1145/3468264.3468548.
[32] Ernst M D, Perkins J H, Guo P J, McCamant S, Pacheco C, Tschantz M S, Xiao C. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, 2007, 69(1/2/3): 35-45. DOI: /10.1016/j.scico.2007.01.015.
[33] Jiang H, Elbaum S, Detweiler C. Inferring and monitoring invariants in robotic systems. Autonomous Robots, 2017, 41(4): 1027-1046. DOI: 10.1007/s10514-016-9576-y.
[34] Bai X, Yang M, Shi B, Liao M. Deep learning for scene text detection and recognition. SCIENTIA SINICA Informationis, 2018, 48(5): 531-544. DOI: 10.1360/N112018-00003. (in Chinese)
[35] Matalonga S, Travassos G H. Testing context-aware software systems: Unchain the context, set it free! In Proc. the 31st Brazilian Symposium on Software Engineering, September 2017, pp.250-254. DOI: 10.1145/3131151.3131190.
[36] Dourish P. What we talk about when we talk about context. Personal and Ubiquitous Computing, 2004, 8(1): 19-30. DOI: 10.1007/s00779-003-0253-8.
[37] Ramirez A J, Jensen A C, Cheng B H C. A taxonomy of uncertainty for dynamically adaptive systems. In Proc. the 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, June 2012, pp.99-108. DOI: 10.1109/SEAMS.2012.6224396.
[38] Yang W, Xu C, Liu Y, Cao C, Ma X, Lu J. Verifying self-adaptive applications suffering uncertainty. In Proc. the 29th ACM/IEEE International Conference on Automated Software Engineering, September 2014, pp.199-210. DOI: 10.1145/2642937.2642999.
[39] Wohlin C, Runeson P, Höst M, Ohlsson M C, Regnell B, Wessl\'{e}n A. Experimentation in Software Engineering. Springer, 2000. DOI: 10.1007/978-3-642-29044-2.
[40] Zhang L, Xu C, Ma X, Gu T, Hong X, Cao C, Lu J. Resynchronizing model-based self-adaptive systems with environments. In Proc. the 19th Asia-Pacific Software Engineering Conference, December 2012, pp.184-193. DOI: 10.1109/APSEC.2012.62.
[41] Dey A K, Abowd G D, Salber D. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-Computer Interaction, 2001, 16(2): 97-166. DOI: 10.1207/S15327051HCI16234-02.
[42] Magnusson S. Similarities and differences in psychophysiological reactions between simulated and real air-to-ground missions. The International Journal of Aviation Psychology, 2002, 12(1): 49-61. DOI: 10.1207/S15327108IJAP1201-5.
[43] Bishop I D, Rohrmann B. Subjective responses to simulated and real environments: A comparison. Landscape and Urban Planning, 2003, 65(4): 261-277. DOI: 10.1016/S0169-2046(03)00070-7.
[44] Qin Y, Xu C, Yu P, Lu J. SIT: Sampling-based interactive testing for self-adaptive apps. Journal of Systems and Software, 2016, 120: 70-88. DOI: 10.1016/j.jss.2016.07.002.
[45] Weyns D, Schmerl B, Grassi V, Malek S, Mirandola R, Prehofer C, Wuttke J, Andersson J, Giese H, Göschka K M. On patterns for decentralized control in self-adaptive systems. Software Engineering for Self-Adaptive Systems II, 2013, pp.76-107. DOI: 10.1007/978-3-642-35813-5-4.
[46] Kephart J O, Chess D M. The vision of autonomic computing. Computer, 2003, 36(1): 41-50. DOI: 10.1109/MC.2003.1160055.
[1] Xu-Zhou Zhang, Yun-Zhan Gong, Ya-Wen Wang, Ying Xing, Ming-Zhe Zhang. 对于包含字符串以及字符串函数的程序的自动化约束求解方法[J]. , 2017, 32(6): 1125-1135.
[2] Xiangyu Zhang, Dongmei Zhang, Yves Le Traon, Qing Wang, Lu Zhang. 圆桌讨论:新兴软件系统的研究机遇与挑战[J]. , 2015, 30(5): 935-941.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] . Online First Under Construction [J]. 计算机科学技术学报, 0, (): 1 .
[2] Zhi-Neng Chen, Chong-Wah Ngo, Wei Zhang, Juan Cao, Yu-Gang Jiang. 网络视频人脸—姓名关联：大规模数据库，基准实验和开放性问题[J]. , 2014, 29(5): 785 -798 .
[3] Fei Xia, De-Jun Jiang, Jin Xiong, Ning-Hui Sun. PCM内存系统研究综述[J]. , 2015, 30(1): 121 -144 .
[4] André Brinkmann, Kathryn Mohror, Weikuan Yu, Philip Carns, Toni Cortes, Scott A. Klasky, Alberto Miranda, Franz-Josef Pfreundt, Robert B. Ross, Marc-André Vef. 高性能计算专用文件系统[J]. 计算机科学技术学报, 2020, 35(1): 4 -26 .
[5] Yu-Tong Lu, Peng Cheng, Zhi-Guang Chen. Tianhe-2数据存储与管理系统设计与实现[J]. 计算机科学技术学报, 2020, 35(1): 27 -46 .
[6] Reza Jafari Ziarani, Reza Ravanmehr. 推荐系统中的意外效应：系统文献综述[J]. 计算机科学技术学报, 2021, 36(2): 375 -396 .
[7] Bo-Han Li, Yi Liu, An-Man Zhang, Wen-Huan Wang, Shuo Wan. 实体消解中分块技术的综述[J]. 计算机科学技术学报, 2020, 35(4): 769 -793 .
[8] Lie-Huang Zhu, Bao-Kun Zheng, Meng Shen, Feng Gao, Hong-Yu Li, Ke-Xin Shi. 比特币系统的数据安全和隐私综述[J]. 计算机科学技术学报, 2020, 35(4): 843 -862 .
[9] 梁盾, 郭元晨, 张少魁, 穆太江, 黄晓蕾. 车道检测-新结果和调查研究[J]. 计算机科学技术学报, 2020, 35(3): 493 -505 .
[10] Lan Huang, Da-Lin Li, Kang-Ping Wang, Teng Gao, Adriano Tavares. 一个关于高级综合工具性能优化的综述[J]. 计算机科学技术学报, 2020, 35(3): 697 -720 .
版权所有 © 《计算机科学技术学报》编辑部
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持：support@magtech.com.cn
总访问量：