HmTest: 基于模型驱动和强化学习的鸿蒙应用自动化测试框架
HmTest: Automated Testing of HarmonyOS Apps via Model-Driven Navigation and Reinforcement Learning
-
摘要:研究背景 鸿蒙操作系统（HarmonyOS）是一款面向智能设备的全场景操作系统，其软件生态正在迅速扩展，截至2024年10月，鸿蒙操作系统上已有超过15,000个原生鸿蒙应用。然而，随着其生态规模的不断扩张，如何对鸿蒙应用进行有效的自动化测试以保障鸿蒙应用的软件质量，已成为亟待解决的重要问题。然而，现有的鸿蒙应用测试工具在页面覆盖率和语句覆盖率等方面均存在不足，因而迫切需要一种更高效的自动化测试方法。目的 本论文旨在提出一套面向原生鸿蒙应用的自动化测试框架——HmTest。该框架提出一种融合了模型驱动和强化学习的探索策略，分别用于实现对应用页面与其内部状态的快速覆盖，从而大幅提升鸿蒙应用自动化测试的效率和效果。方法 HmTest框架主要包含两个模块：Targeted Exploration模块和RL-based Exploration模块。Targeted Exploration模块采用白盒测试方法，通过对鸿蒙应用源代码进行静态分析，构建页面跳转图（PTG），利用该图对测试流程进行有针对性的跳转，进而快速实现较高的页面覆盖率。RL-based Exploration模块则采用黑盒测试方法，利用强化学习算法指导测试过程，深入探索应用的整个状态空间，并在遇到探索停滞时，借助基于自动机的自动恢复机制高效重启测试过程，确保测试的连续性与稳定性，进而达到更高的语句覆盖率。结果 我们在九个HarmonyOS Next应用上对HmTest进行了综合评估，其中八个为开源应用。实验结果表明， Targeted Exploration模块能够根据鸿蒙应用源代码生成高质量的页面跳转图，并在短时间内迅速覆盖应用的大部分页面，而RL-based Exploration模块则能够达到更高的语句覆盖率，在大多数应用中显著优于官方提供的测试工具wukong和DevEco Testing。结论 本论文介绍的HmTest鸿蒙应用自动化测试框架提出基于模型驱动和强化学习的探索策略，分别用于实现对应用页面和内部状态的快速覆盖。实验结果表明，在大多数应用中，Targeted Exploration模块和RL-based Exploration模块分别在页面覆盖率和语句覆盖率显著优于现有的官方测试工具。未来，我们将致力于引入深度强化学习和多模态大模型等前沿的技术，以进一步提高鸿蒙应用自动化测试的效率和效果。Abstract: HarmonyOS is a new all-scenario operating system for smart devices. As its software ecosystem expands rapidly, how to conduct automated testing of HarmonyOS apps for quality assurance has become a crucial task. This paper presents HmTest, an automated testing framework for HarmonyOS apps, which consists of two complementary modules: targeted exploration and reinforcement learning (RL)-based exploration. Targeted exploration performs white-box testing, leveraging static analysis to construct a page transition graph (PTG). By systematically traversing PTGs to guide testing, HmTest can quickly achieve high page coverages. On the other hand, RL-based exploration performs black-box testing, utilizing reinforcement learning to achieve a comprehensive exploration of app states. Additionally, an automaton-based mechanism is employed to efficiently recover and restart the testing process when reinforcement learning encounters stagnation. We have evaluated HmTest on nine HarmonyOS NEXT apps and compared it with two official HarmonyOS app testing tools. The experimental results demonstrate that targeted exploration can generate highly-precise PTGs and help achieve high page coverages within a few minutes. RL-based exploration can significantly outperform other methods in terms of finer-grained statement coverage on the majority of the tested apps and benefits from the recovery mechanism. To facilitate future research, we have made HmTest open-source at https://github.com/sqlab-sustech/hmtest and provided a video demo at https://jcst.ict.ac.cn/news/361.
下载: