›› 2014, Vol. 29 ›› Issue (4): 713-723.doi: 10.1007/s11390-014-1461-6

Special Issue: Software Systems

• Software Systems • Previous Articles     Next Articles

Cooperative Software Testing and Analysis:Advances and Challenges

Tao Xie1 (谢涛), Senior Member, ACM, IEEE, Lu Zhang2,3 (张路), Senior Member, CCF, Member, ACM, Xusheng Xiao4 (肖旭生), Ying-Fei Xiong2,3 (熊英飞), Member, CCF, ACM, IEEE, and Dan Hao2,3 (郝丹), Member, CCF, ACM, IEEE   

  1. 1. Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, U.S.A.;
    2. Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing 100871, China;
    3. Institute of Software, School of Electronics Engineering and Computer Science, Peking University, Beijing 100871, China;
    4. NEC Laboratories America, Inc., Princeton, NJ 08520, U.S.A.
  • Online:2014-07-05 Published:2014-07-05
  • About author:Tao Xie is an associate professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign, USA. He received his Ph.D. degree in computer science from the University of Washington in 2005. Before that, he received an M.S. degree in computer science from the University of Washington in 2002 and an M.S. degree in computer science from Peking University in 2000. His research interests are in software engineering, with an emphasis on software testing, program analysis, and software analytics.
  • Supported by:

    This work is supported in part by the National Natural Science Foundation of China under Grant Nos. 61228203, 61225007, and 61272157, the National Science Foundation of USA under Grant Nos. CCF-1349666, CNS-1434582, CCF-1434596, CCF-1434590, CNS-1439481, and a Microsoft Research award.

In recent years, to maximize the value of software testing and analysis, we have proposed the methodology of cooperative software testing and analysis (in short as cooperative testing and analysis) to enable testing and analysis tools to cooperate with their users (in the form of tool-human cooperation), and enable one tool to cooperate with another tool (in the form of tool-tool cooperation). Such cooperations are motivated by the observation that a tool is typically not powerful enough to address complications in testing or analysis of complex real-world software, and the tool user or another tool may be able to help out some problems faced by the tool. To enable tool-human or tool-tool cooperation, effective mechanisms need to be developed (1) for a tool to communicate problems faced by the tool to the tool user or another tool, (2) for the tool user or another tool to assist the tool to address the problems. Such methodology of cooperative testing and analysis forms a new research frontier on synergistic cooperations between humans and tools along with cooperations between tools and tools. This article presents recent example advances and challenges on cooperative testing and analysis.

[1] Xie T, Tillmann N, de Halleux J, Schulte W. Future of developer testing: Building quality in code. In Proc. the 18th FSE/SDP Workshop on the Future of Software Engineering Research, Nov. 2010, pp.415-420.

[2] Xiao X, Thummalapenta S, Xie T. Advances on improving automation in developer testing. In Advances in Computers, volume 85, Memon A (ed.), Burlington: Academic Press, 2012, pp.165-212.

[3] D'Silva V, Kroening D, Weissenbacher G. A survey of automated techniques for formal software verification. IEEE Trans. Computer-Aided Design of Integrated Circuits and Systems, 2008, 27(7): 1165-1178.

[4] Xiao X, Xie T, Tillmann N, de Halleux J. Precise identification of problems for structural test generation. In Proc. the 33rd Int. Conf. Software Engineering, May 2011, pp.611-620.

[5] Dillig I, Dillig T, Aiken A. Automated error diagnosis using abductive inference. In Proc. the 33rd ACM SIGPLAN Conf. Programming Language Design and Implementation, June 2012, pp.181-192.

[6] Xie T. Cooperative testing and analysis: Human-tool, tooltool, and human-human cooperations to get work done. In Proc. the 12th Int. Working Conf. Source Code Analysis and Manipulation, Sept. 2012, pp.1-3.

[7] Hellerstein J L, Diao Y, Parekh S, Tilbury D M. Feedback Control of Computing Systems. John Wiley & Sons, 2004.

[8] Godefroid P, Klarlund N, Sen K. DART: Directed automated random testing. In Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, June 2005, pp.213-223.

[9] Tillmann N, de Halleux J. Pex-white box test generation for .NET. In Proc. the 2nd Int. Conf. Tests and Proofs, April 2008, pp.134-153.

[10] Tillmann N, Schulte W. Mock-object generation with behavior. In Proc. the 21st Int. Conf. Automated Software Engineering, Sept. 2006, pp.365-368.

[11] Marri M R, Xie T, Tillmann N, de Halleux J, Schulte W. An empirical study of testing file-system-dependent software with mock objects. In Proc. the 4th Int. Workshop Automation of Software Test, May 2009, pp.149-153.

[12] Taneja K, Zhang Y, Xie T. MODA: Automated test generation for database applications via mock objects. In Proc. the 25th Int. Conf. Automated Software Engineering, Sept. 2010, pp.289-292.

[13] St-Amour V, Tobin-Hochstadt S, Felleisen M. Optimization coaching: Optimizers learn to communicate with programmers. In Proc. the 3rd ACM SIGPLAN Int. Conf. ObjectOriented Programming, Systems, Languages, and Applications, Oct. 2012, pp.163-178.

[14] Dincklage D, Diwan A. Explaining failures of program analyses. In Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, June 2008, pp.260-269.

[15] Lakhotia K, McMinn P, Harman M. An empirical investigation into branch coverage for C programs using CUTE and AUSTIN. Journal of Systems and Software, 2010, 83(12): 2379-2391.

[16] Godefroid P, Levin M Y, Molnar D A. Automated whitebox fuzz testing. In Proc. Network and Distributed System Security Symp., Feb. 2008.

[17] Xie T, Tillmann N, de Halleux P, Schulte W. Fitness-guided path exploration in dynamic symbolic execution. In Proc. the 39th IEEE/IFIP Int. Conf. Dependable Systems and Networks, July 2009, pp.359-368.

[18] Godefroid P, Luchaup D. Automatic partial loop summarization in dynamic test generation. In Proc. Int. Symp. Software Testing and Analysis, July 2011, pp.23-33.

[19] Saxena P, Poosankam P, McCamant S, Song D. Loopextended symbolic execution on binary programs. In Proc. the 18th Int. Symp. Software Testing and Analysis, July 2009, pp.225-236. 722 J. Comput. Sci. & Technol., July 2014, Vol.29, No.4

[20] Strej?ek J, TrtÍk M. Abstracting path conditions. In Proc. Int. Symp. Software Testing and Analysis, July 2012, pp.155165.

[21] Xiao X, Li S, Xie T, Tillmann N. Characteristic studies of loop problems for structural test generation via symbolic execution. In Proc. the 28th Int. Conf. Automated Software Engineering, Nov. 2013, pp.246-256.

[22] Tillmann N, de Halleux J, Xie T. Pex4Fun: Teaching and learning computer science via social gaming. In Proc. the 24th IEEE-CS Conf. Software Engineering Education and Training, May 2011, pp.546-548.

[23] Tillmann N, de Halleux J, Xie T, Gulwani S, Bishop J. Teaching and learning programming and software engineering via interactive gaming. In Proc. the 35th Int. Conf. Software Engineering, Software Engineering Education, May 2013, pp.1117-1126.

[24] Tillmann N, de Halleux J, Xie T, Bishop J. Pex4Fun: A webbased environment for educational gaming via automated test generation. In Proc. the 28th Int. Conf. Automated Software Engineering, Nov. 2013, pp.730-733.

[25] Hao D, Zhang L, Xie T, Mei H, Sun J. Interactive fault localization using test information. Journal of Computer Science and Technology, 2009, 24(5): 962-974.

[26] Hao D, Zhang L, Zhang L, Sun J, Mei H. VIDA: Visual interactive debugging. In Proc. the 31st Int. Conf. Software Engineering, May 2009, pp.583-586.

[27] Xie T, Tillmann N, de Halleux J. Educational software engineering: Where software engineering, education, and gaming meet. In Proc. the 3rd International Workshop on Games and Software Engineering, May 2013, pp.36-39.

[28] Fincher S, Cooper S, Kölling M, Maloney J. Comparing Alice, Greenfoot & Scratch. In Proc. the 41st ACM Technical Symp. Computer Science Education, March 2010, pp.192193.

[29] Utting I, Cooper S, Kölling M, Maloney J, Resnick M. Alice, Greenfoot, and Scratch-A discussion. ACM Trans. Computing Education, 2010, 10(4): 17:1-17:11.

[30] Tillmann N, de Halleux J, Xie T, Bishop J. Code Hunt: Gamifying teaching and learning of computer science at scale. In Proc. the 1st ACM Conf. Learning at Scale, March 2014, pp.221-222.

[31] Tillmann N, Bishop J, Horspool N, Perelman D, Xie T. Code Hunt Searching for secret code for fun. In Proc. the 7th Int. Workshop Search-Based Software Testing, June 2014.

[32] Dwyer M B, Elbaum S G. Unifying verification and validation techniques: Relating behavior and properties through partial evidence. In Proc. the 18th FSE/SDP Workshop on the Future of Software Engineering Research, Nov. 2010, pp.93-98.

[33] Ge X, Taneja K, Xie T, Tillmann N. DyTa: Dynamic symbolic execution guided with static verification results. In Proc. the 33rd Int. Conf. Software Engineering, May 2011, pp.992-994.

[34] Xie T, Notkin D. Tool-assisted unit-test generation and selection based on operational abstractions. Automated Software Engineering, 2006, 13(3): 345-371.

[35] Wang X, Zhang L, Xie T, Xiong Y, Mei H. Automating presentation changes in dynamic web applications via collaborative hybrid analysis. In Proc. the 20th ACM SIGSOFT Symp. Foundations of Software Engineering, Nov. 2012, pp.16:1-16:11.

[36] Csallner C, Smaragdakis Y, Xie T. DSD-Crasher: A hybrid analysis tool for bug finding. ACM Trans. Software Engineering and Methodology, 2008, 17(2): 8:1-8:37.

[37] Thummalapenta S, Xie T, Tillmann N, de Halleux J, Su Z. Synthesizing method sequences for high-coverage testing. In Proc. ACM SIGPLAN Int. Conf. Object-Oriented Programming, Systems, Languages, and Applications, Oct. 2011, pp.189-206.

[38] Inkumsah K, Xie T. Improving structural testing of objectoriented programs via integrating evolutionary testing and symbolic execution. In Proc. the 23rd Int. Conf. Automated Software Engineering, Sept. 2008, pp.297-306.

[39] Xiong Y, Liu D, Hu Z, Zhao H, Takeichi M, Mei H. Towards automatic model synchronization from model transformations. In Proc. the 22nd Int. Conf. Automated Software Engineering, Nov. 2007, pp.164-173.

[40] Weiser M. Program slicing. In Proc. the 5th Int. Conf. Software Engineering, March 1981, pp.439-449.

[41] Gong L, Lo D, Jiang L, Zhang H. Interactive fault localization leveraging simple user feedback. In Proc. the 28th IEEE Int. Conf. Software Maintenance, Sept. 2012, pp.67-76.

[42] Weimer W, Nguyen T, Le Goues C, Forrest S. Automatically finding patches using genetic programming. In Proc. the 31st Int. Conf. Software Engineering, May 2009, pp.364-374.
No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] Meng Liming; Xu Xiaofei; Chang Huiyou; Chen Guangxi; Hu Mingzeng; Li Sheng;. A Tree-Structured Database Machine for Large Relational Database Systems[J]. , 1987, 2(4): 265 -275 .
[2] Lin Shan;. Using a Student Model to Improve Explanation in an ITS[J]. , 1992, 7(1): 92 -96 .
[3] Fei Xianglin; Liao Lei; Wang Hezhen; Wang Chengzao;. Structured Development Environment Based on the Object-Oriented Concepts[J]. , 1992, 7(3): 193 -201 .
[4] Shen Yidong;. Form alizing Incomplete Knowledge in Incomplete Databases[J]. , 1992, 7(4): 295 -304 .
[5] wang Xuejun; Shi Chunyi;. A Multiagent Dynamic interaction Testbed:Theoretic Framework, System Architecture and Experimentation[J]. , 1997, 12(2): 121 -132 .
[6] LUO Junzhou; GU Guanqun; FEI Xiang;. An Architectural Model for Intelligent Network Management[J]. , 2000, 15(2): 136 -143 .
[7] NIE Xumin; GUO Qing;. Renaming a Set of Non-Horn Clauses[J]. , 2000, 15(5): 409 -415 .
[8] Hua Li, Shui-Cheng Yan, and Li-Zhong Peng[1]. Robust Non-Frontal Face Alignment with Edge Based Texture[J]. , 2005, 20(6): 849 -854 .
[9] Feng Yi, Qi-Chao Sun, Jie Dong, and Lu Yu. Low-Complexity Tools in AVS Part 7[J]. , 2006, 21(3): 345 -353 .
[10] Fei-Long Tang, Ming-Lu Li, and Joshua Zhexue Huang. Automatic Transaction Compensation for Reliable Grid Applications[J]. , 2006, 21(4): 529 -536 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

Home
Editorial Board
Author Guidelines
Subscription
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved