Journal of Computer Science and Technology ›› 2023, Vol. 38 ›› Issue (2): 455-470.doi: 10.1007/s11390-022-0824-7

Special Issue: Software Systems; Computer Networks and Distributed Computing

• Regular Paper • Previous Articles    

Optimization of Web Service Testing Task Assignment in Crowdtesting Environment

Wen-Jun Tang (唐文君), Rong Chen* (陈　荣), Member, CCF, ACM, IEEE, Jia-Li Zhang (张佳丽), Lin Huang (黄　琳), Sheng-Jie Zheng (郑圣杰), and Shi-Kai Guo (郭世凯), Member, CCF   

  1. Information Science and Technology College, Dalian Maritime University, Dalian 116026, China
  • Received:2020-07-22 Revised:2022-11-30 Accepted:2022-12-23 Online:2023-05-10 Published:2023-05-10
  • Contact: Rong Chen E-mail:rchen@dlmu.edu.cn
  • About author:Rong Chen received his M.S. and Ph.D. degrees in computer software and theory from Jilin University, Changchun, in 1997 and 2000, respectively. He is currently a professor of the Information Science and Technology College at the Dalian Maritime University, Dalian, and has previously held position at Sun Yat-sen University, Guangzhou. His research interests include software diagnosis, collective intelligence, activity recognition, Internet and mobile computing. He is a member of CCF, ACM and IEEE.
  • Supported by:
    This work was supported by the National Natural Science Foundation of China under Grant Nos. 61672122, 61902050 and 61602077, the Fundamental Research Funds for the Central Universities of China under Grant No. 3132019355, and the CERNET Innovation Project under Grant No. NGII20190627.

PDF

41

Supplement

1

Crowdtesting has emerged as an attractive and economical testing paradigm that features testers from different countries, with various backgrounds and working conditions. Recent developments in crowdsourcing testing suggest that it is feasible to manage test populations and processes, but they are often outside the scope of standard testing theory. This paper explores how to allocate service-testing tasks to proper testers in an ever-changing crowdsourcing environment. We formalize it as an optimization problem with the objective to ensure the testing quality of the crowds, while considering influencing factors such as knowledge capability, the rewards, the network connections, and the geography and the skills required. To solve the proposed problem, we design a task assignment algorithm based on the Differential Evolution (DE) algorithm. Extensive experiments are conducted to evaluate the efficiency and effectiveness of the proposed algorithm in real and synthetic data, and the results show better performance compared with other heuristic-based algorithms.

Key words: crowdtesting; task assignment; web service testing; heuristic algorithm; optimization; quality of web service;

[1] Hussain S, Wang Z S, Toure I K, Diop A. Web service testing tools: A comparative study. International Journal of Computer Science Issues, 2013, 10(1/2/3): 641–647.
[2] Yu H, Shen Z Q, Fauvel S, Cui L Z. Efficient scheduling in crowdsourcing based on workers' mood. In Proc. the 2017 IEEE Int. Conf.  Agents (ICA), Jul. 2017, pp.121–126. DOI: 10.1109/AGENTS.2017.8015317.
[3] Rahman H, Roy S B, Thirumuruganathan S, Amer-Yahia S, Das G. Task assignment optimization in collaborative crowdsourcing. In Proc. the 2015 IEEE Int. Conf.  Data Mining, Nov. 2015, pp.949–954. DOI: 10.1109/ICDM.2015.119.
[4] Komarov S, Reinecke K, Gajos K Z. Crowdsourcing performance evaluations of user interfaces. In Proc. the 2013 SIGCHI Conf.  Human Factors in Computing Systems, Apr. 2013, pp.207–216. DOI: 10.1145/2470654.2470684.
[5] Surowiecki J. The Wisdom of Crowds. Anchor Books, 2005.
[6] Yan M Z, Sun H L, Liu X D. iTest: Testing software with mobile crowdsourcing. In Proc. the 1st Int. Workshop on Crowd-Based Software Development Methods and Technologies, Nov. 2014, pp.19–24. DOI: 10.1145/2666539.2666569.
[7] Yan M Z, Sun H L, Liu X D. Efficient testing of web services with mobile crowdsourcing. In Proc. the 7th Asia-Pacific Symp.  Internetware, Nov. 2015, pp.157–165. DOI: 10.1145/2875913.2875926.
[8] Chen R, Guo S K, Wang X Z, Zhang T L. Fusion of multi-RSMOTE with fuzzy integral to classify bug reports with an imbalanced distribution. IEEE Trans. Fuzzy Systems, 2019, 27(12): 2406–2420. DOI: 10.1109/TFUZZ.2019.2899809.
[9] Jiang H, Li X C, Ren Z L, Xuan J F, Jin Z. Toward better summarizing bug reports with crowdsourcing elicited attributes. IEEE Trans. Reliability, 2019, 68(1): 2–22. DOI: 10.1109/TR.2018.2873427.
[10] Liang W, Yu Z W, Qi H, Guo B, Xiong H Y. Multi-objective optimization based allocation of heterogeneous spatial crowdsourcing tasks. IEEE Trans. Mobile Computing, 2018, 17(7): 1637–1650. DOI: 10.1109/TMC.2017.2771259.
[11] Roy S B, Lykourentzou I, Thirumuruganathan S, Amer-Yahia S, Das G. Task assignment optimization in knowledge-intensive crowdsourcing. The VLDB Journal, 2015, 24(4): 467–491. DOI: 10.1007/s00778-015-0385-2.
[12] Catallo I, Martinenghi D. The dimensions of crowdsourcing task design. In Proc. the 17th Int. Conf. Web Engineering (ICWE), Jun. 2017, pp.394–402. DOI: 10.1007/978-3-319-60131-1_25.
[13] Zhu H, Zhang Y F. Collaborative testing of web services. IEEE Trans. Services Computing, 2012, 5(1): 116–130. DOI: 10.1109/TSC.2010.54.
[14] Guo S K, Chen R, Li H, Zhang T L, Liu Y Q. Identify severity bug report with distribution imbalance by CR-SMOTE and ELM. International Journal of Software Engineering and Knowledge Engineering, 2019, 29(2): 139–175. DOI: 10.1142/S0218194019500074.
[15] Gao L, Gan Y, Zhou B H, Dong M Y. A user-knowledge crowdsourcing task assignment model and heuristic algorithm for Expert Knowledge Recommendation Systems. Engineering Applications of Artificial Intelligence, 2020, 96: 103959. DOI: 10.1016/j.engappai.2020.103959.
[16] Zou D X, Liu H K, Gao L Q, Li S. An improved differential evolution algorithm for the task assignment problem. Engineering Applications of Artificial Intelligence, 2011, 24(4): 616–624. DOI: 10.1016/j.engappai.2010.12.002.
[17] Rahman H, Thirumuruganathan S, Roy S B, Amer-Yahia S, Das G. Worker skill estimation in team-based tasks. Proceedings of the VLDB Endowment, 2015, 8(11): 1142–1153. DOI: 10.14778/2809974.2809977.
[18] Khazankin R, Satzger B, Dustdar S. Optimized ution of business processes on crowdsourcing platforms. In Proc. the 8th Int. Conf.  Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom), Oct. 2012, pp.443–451. DOI: 10.4108/icst.collaboratecom.2012.250434.
[19] Anagnostopoulos A, Becchetti L, Castillo C, Gionis A, Leonardi S. Online team formation in social networks. In Proc. the 21st Int. Conf.  World Wide Web, Apr. 2012, pp.839–848. DOI: 10.1145/2187836.2187950.
[20] Deng W, Zhao H M, Yang X H, Xiong J X, Sun M, Li B. Study on an improved adaptive PSO algorithm for solving multi-objective gate assignment. Applied Soft Computing, 2017, 59: 288–302. DOI: 10.1016/j.asoc.2017.06.004.
[21] Deng W, Xu J J, Zhao H M. An improved ant colony optimization algorithm based on hybrid strategies for scheduling problem. IEEE Access, 2019, 7: 20281–20292. DOI: 10.1109/ACCESS.2019.2897580.
[22] Karaboğa D, Ökdem S. A simple and global optimization algorithm for engineering problems: Differential evolution algorithm. Turkish Journal of Electrical Engineering and Computer Sciences, 2004, 12(1): 53–60.
[23] Anagnostopoulos A, Becchetti L, Fazzone A, Mele I, Riondato M. The importance of being expert: Efficient max-finding in crowdsourcing. In Proc. the 2015 ACM SIGMOD Int. Conf. Management of Data, May 2015, pp.983–998. DOI: 10.1145/2723372.2723722.
[24] Tran-Thanh L, Stein S, Rogers A, Jennings N R. Efficient crowdsourcing of unknown experts using bounded multi-armed bandits. Artificial Intelligence, 2014, 214: 89–111. DOI: 10.1016/j.artint.2014.04.005.
[25] Yang D J, Xue G L, Fang X, Tang J. Incentive mechanisms for crowdsensing: Crowdsourcing with smartphones. IEEE/ACM Trans. Networking, 2016, 24(3): 1732–1744. DOI: 10.1109/TNET.2015.2421897.
[26] Zheng Z B, Zhang Y L, Lyu M R. Investigating QoS of real-world web services. IEEE Trans. Services Computing, 2014, 7(1): 32–39. DOI: 10.1109/TSC.2012.34.
[27] Feige U, Mirrokni V S, Vondrák J. Maximizing non-monotone submodular functions. SIAM Journal on Computing, 2011, 40(4): 1133–1153. DOI: 10.1137/090779346.
[28] Tian X T, Li H H, Liu F. Web service reliability test method based on log analysis. In Proc. the 2017 IEEE Int. Conf. Software Quality, Reliability and Security Companion (QRS-C), Jul. 2017, pp.195–199. DOI: 10.1109/QRS-C.2017.38.
[29] Gardlo B, Egger S, Seufert M, Schatz R. Crowdsourcing 2.0: Enhancing ution speed and reliability of web-based QoE testing. In Proc. the 2014 IEEE Int. Conf. Communications (ICC), Jun. 2014, pp.1070–1075. DOI: 10.1109/ICC.2014.6883463.
[30] Gardlo B. Quality of experience evaluation methodology via crowdsourcing [Ph.D. thesis]. University of Žilina, Žilina, 2012.
[31] Blanco R, Halpin H, Herzig D M, Mika P, Pound J, Thompson H S, Duc T T. Repeatable and reliable search system evaluation using crowdsourcing. In Proc. the 34th Int. ACM SIGIR Conf. Research and Development in Information Retrieval, Jul. 2011, pp.923–932. DOI: 10.1145/2009916.2010039.
[32] Sherief N, Jiang N, Hosseini M, Phalp K, Ali R. Crowdsourcing software evaluation. In Proc. the 18th Int. Conf. Evaluation and Assessment in Software Engineering, May 2014, pp.19. DOI: 10.1145/2601248.2601300.
[33] Chen F X, Kim S. Crowd debugging. In Proc. the 10th Joint Meeting on Foundations of Software Engineering, Aug. 2015, pp.320–332. DOI: 10.1145/2786805.2786819.
[34] Petrillo F, Lacerda G, Pimenta M, Freitas C. Visualizing interactive and shared debugging sessions. In Proc. the  3rd IEEE Working Conf.  Software Visualization (VISSOFT), Sept. 2015, pp.140–144. DOI: 10.1109/VISSOFT.2015.7332425.
[35] Petrillo F, Soh Z, Khomh F, Pimenta M, Freitas C, Guéhéneuc Y. Towards understanding interactive debugging. In Proc. the 2016 IEEE Int. Conf. Software Quality, Reliability and Security (QRS), Aug. 2016, pp.152–163. DOI: 10.1109/QRS.2016.27.
[36] Chen X, Jiang H, Chen Z Y, He T K, Nie L M. Automatic test report augmentation to assist crowdsourced testing. Frontiers of Computer Science, 2019, 13(5): 943–959. DOI: 10.1007/s11704-018-7308-5.
[37] Guaiani F, Muccini H. Crowd and laboratory testing, can they co-exist? An exploratory study. In Proc. the 2nd IEEE/ACM Int. Workshop on CrowdSourcing in Software Engineering, May 2015, pp.32–37. DOI: 10.1109/CSI-SE.2015.14.
[38] Stol K J, Fitzgerald B. Research protocol for a case study of crowdsourcing software development. Technical Report, TR_2014_03, Lero, 2014. DOI: 10.13140/2.1.1151.3123. https://www.researchgate.net/publication/273383598_Research_Protocol_for_a_Case_Study_of_Crowdsourcing_Software_Development, Mar. 2023.
[39] Jiang H, Nie L M, Sun Z Y, Ren Z L, Kong W Q, Zhang T, Luo X P. ROSF: Leveraging information retrieval and supervised learning for recommending code snippets. IEEE Trans. Services Computing, 2019, 12(1): 34–46. DOI: 10.1109/TSC.2016.2592909.
[40] Boutsis I, Kalogeraki V. Crowdsourcing under real-time constraints. In Proc. the 27th Int. Symp. Parallel and Distributed Processing, May 2013, pp.753–764. DOI: 10.1109/IPDPS.2013.84.
[41] Boutsis I, Kalogeraki V. On task assignment for real-time reliable crowdsourcing. In Proc. the 34th Int. Conf. Distributed Computing Systems, Jul. 2014. DOI: 10.1109/ICDCS.2014.9.
[1] Xian-He Sun and Xiaoyang Lu. The Memory-Bounded Speedup Model and Its Impacts in Computing [J]. Journal of Computer Science and Technology, 2023, 38(1): 64-79.
[2] Zi-Xuan Hu, Peng-Bo Bo, and Cai-Ming Zhang. Quasi-Developable B-Spline Surface Design with Control Rulings [J]. Journal of Computer Science and Technology, 2022, 37(5): 1221-1238.
[3] Abdalaziz Sawwan and Jie Wu. Energy-Efficient Minimum Mobile Charger Coverage for Wireless Sensor Networks [J]. Journal of Computer Science and Technology, 2022, 37(4): 869-887.
[4] Que-Ping Kong, Zi-Yan Wang, Yuan Huang, Xiang-Ping Chen, Xiao-Cong Zhou, Zi-Bin Zheng, and Gang Huang. Characterizing and Detecting Gas-Inefficient Patterns in Smart Contracts [J]. Journal of Computer Science and Technology, 2022, 37(1): 67-82.
[5] Yu-Wei Wu, Qing-Gang Wang, Long Zheng, Xiao-Fei Liao, Hai Jin, Wen-Bin Jiang, Ran Zheng, Kan Hu. FDGLib: A Communication Library for Efficient Large-Scale Graph Processing in FPGA-Accelerated Data Centers [J]. Journal of Computer Science and Technology, 2021, 36(5): 1051-1070.
[6] Xiao-Jing Zha, Yin-Shui Xia, Shang-Luan Xie, Zhu-Fei Chu. Defect-Tolerant Mapping of CMOL Circuit Targeting Delay Optimization [J]. Journal of Computer Science and Technology, 2021, 36(5): 1118-1132.
[7] Hui-Ming Tian, Zhu-Fei Chu. Inversion Optimization Strategy Based on Primitives with Complement Attributes [J]. Journal of Computer Science and Technology, 2021, 36(5): 1145-1154.
[8] Yu-Jie Yuan, Yukun Lai, Tong Wu, Lin Gao, Li-Gang Liu. A Revisit of Shape Editing Techniques: From the Geometric to the Neural Viewpoint [J]. Journal of Computer Science and Technology, 2021, 36(3): 520-554.
[9] Jun Gao, Paul Liu, Guang-Di Liu, Le Zhang. Robust Needle Localization and Enhancement Algorithm for Ultrasound by Deep Learning and Beam Steering Methods [J]. Journal of Computer Science and Technology, 2021, 36(2): 334-346.
[10] Zeynep Banu Ozger, Nurgul Yuzbasioglu Uslu. An Effective Discrete Artificial Bee Colony Based SPARQL Query Path Optimization by Reordering Triples [J]. Journal of Computer Science and Technology, 2021, 36(2): 445-462.
[11] Jason Liu, Pedro Espina, Xian-He Sun. A Study on Modeling and Optimization of Memory Systems [J]. Journal of Computer Science and Technology, 2021, 36(1): 71-89.
[12] Mohammad Y. Mhawish, Manjari Gupta. Predicting Code Smells and Analysis of Predictions: Using Machine Learning Techniques and Software Metrics [J]. Journal of Computer Science and Technology, 2020, 35(6): 1428-1445.
[13] Bo-Han Li, Yi Liu, An-Man Zhang, Wen-Huan Wang, Shuo Wan. A Survey on Blocking Technology of Entity Resolution [J]. Journal of Computer Science and Technology, 2020, 35(4): 769-793.
[14] Maryam Zarezadeh, Hamid Mala, Homa Khajeh. Preserving Privacy of Software-Defined Networking Policies by Secure Multi-Party Computation [J]. Journal of Computer Science and Technology, 2020, 35(4): 863-874.
[15] Lan Huang, Da-Lin Li, Kang-Ping Wang, Teng Gao, Adriano Tavares. A Survey on Performance Optimization of High-Level Synthesis Tools [J]. Journal of Computer Science and Technology, 2020, 35(3): 697-720.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] Zhou Di;. A Recovery Technique for Distributed Communicating Process Systems[J]. , 1986, 1(2): 34 -43 .
[2] Li Wei;. A Structural Operational Semantics for an Edison Like Language(2)[J]. , 1986, 1(2): 42 -53 .
[3] Chen Shihua;. On the Structure of Finite Automata of Which M Is an(Weak)Inverse with Delay τ[J]. , 1986, 1(2): 54 -59 .
[4] Li Wanxue;. Almost Optimal Dynamic 2-3 Trees[J]. , 1986, 1(2): 60 -71 .
[5] Feng Yulin;. Recursive Implementation of VLSI Circuits[J]. , 1986, 1(2): 72 -82 .
[6] Wang Xuan; Lü Zhimin; Tang Yuhai; Xiang Yang;. A High Resolution Chinese Character Generator[J]. , 1986, 1(2): 1 -14 .
[7] C.Y.Chung; H.R.Hwa;. A Chinese Information Processing System[J]. , 1986, 1(2): 15 -24 .
[8] Sun Zhongxiu; Shang Lujun;. DMODULA:A Distributed Programming Language[J]. , 1986, 1(2): 25 -31 .
[9] Chen Shihua;. On the Structure of (Weak) Inverses of an (Weakly) Invertible Finite Automaton[J]. , 1986, 1(3): 92 -100 .
[10] Gao Qingshi; Zhang Xiang; Yang Shufan; Chen Shuqing;. Vector Computer 757[J]. , 1986, 1(3): 1 -14 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

 Home
Editorial Board
Author Guidelines
Subscription		 Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved