上下文感知系统输入验证方法在模拟环境与物理环境中的对比
Simulation Might Change Your Results: A Comparison of Context-Aware System Input Validation in Simulated and Physical Environments
-
摘要: 1、研究背景
上下文感知系统(Context-aware systems, CAS)融合了软件组件和物理环境。它不断地感知物理环境环境的上下文变化,基于预定义的逻辑做决策,并执行相应的动作以应对环境变化。由于传感器噪声、机械偏差等物理环境中天然存在的问题,CAS通常面临与环境交互过程中的不确定性问题。因此,许多现有的研究工作采用对CAS的输入数据做验证的方法(即输入验证方法)来缓解这一问题。
与传统软件不同,CAS通常在物理环境中运行。然而,根据我们的一项调研,相关文献中只有17%的工作在物理环境中评估了他们的方法,其余工作中提出的方法只在模拟环境中(或只使用预先收集的执行轨迹)评估。根据近期一篇文献综述中的讨论,这可能是因为在物理环境中测试CAS需要付出更加高昂的时间和金钱成本。
2、目的
在这项工作中,我们尝试回答以下两个问题:
(1)CAS输入验证方法在物理环境中的表现与其在模拟环境中的表现是否有差异?
(2)若(1)的结论为“有差异”,那么物理环境和模拟环境的哪些不同点会影响这一差异?
3、方法
我们分别在物理环境(基于大疆RoboMaster机器人小车)和模拟环境(基于Unity引擎)中搭建了无人驾驶汽车测试演示场景,并接入已有工作中提出的三种输入验证方法:约束检测(Constraint checking),不变式检测(Invariant checking)和深度学习模型输入裁剪(Deep learning model input pruning)。我们分别在物理环境和模拟环境中开展相同配置的实验,比较这两种环境中CAS输入验证方法对小车完成任务比例的提高的差异。
4、结果
总体上,物理环境中三种CAS输入验证方法的开启能使小车完成任务的比例提高82%,而在模拟环境中提高的比例为50%。这一结果表明CAS输入验证方法在物理环境和模拟环境中的表现存在差异。细粒度的分析还表明场景设置、物理平台和环境模型等三个因素会对CAS输入验证方法在两种环境中的有效性造成不同程度的影响。
5、结论
CAS输入验证方法在物理环境和模拟环境中的表现可能存在差异,且受场景设置、物理平台和环境模型等因素的影响。因此,面向CAS的方法应该在物理环境中评估,以避免模拟环境带来的偏差。Abstract: Context-aware systems (a.k.a. CASs) integrate cyber and physical space to provide adaptive functionalities in response to changes in context. Building context-aware systems is challenging due to the uncertain running environment. Therefore, many input validation approaches have been proposed to protect context-aware systems from uncertainty and keep them executing safely. However, in contrast to context-aware systems' prevailing in physical environments, most of those academic solutions (83%) are purely evaluated in simulated environments. In this article, we study whether this evaluation setting could lead to biased conclusions. We build a testing platform, RM-Testing, based on DJI RoboMaster robot car, to conduct the physical-environment based experiments. We select three up-to-date input validation approaches, and compare their performance in the simulated environment and in the physical environment. The experimental results show that all three approaches' performance in simulated environments (improving task success rate by 82% compared with the system without the support of input validation) does differ from their performance in a physical environment (improving the task success rate by 50%). We also recognize three factors (scenario setting, physical platform and environmental model) that affect the performance of input validation approaches, based on an execution model of the context-aware system.