Journal of Computer Science and Technology ›› 2019, Vol. 34 ›› Issue (5): 1020-1038.doi: 10.1007/s11390-019-1958-0

Special Issue: Data Management and Data Mining; Software Systems

• Special Section on Software Systems 2019 • Previous Articles     Next Articles

DP-Share: Privacy-Preserving Software Defect Prediction Model Sharing Through Differential Privacy

Xiang Chen1,2,3, Senior Member, CCF, Dun Zhang1, Zhan-Qi Cui2,4, Member, CCF, Qing Gu2, Senior Member, CCF, Xiao-Lin Ju1,2, Member, CCF   

  1. 1 School of Information Science and Technology, Nantong University, Nantong 226019, China;
    2 State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China;
    3 School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798, Singapore;
    4 Computer School, Beijing Information Science and Technology University, Beijing 100101, China
  • Received:2018-11-29 Revised:2019-04-03 Online:2019-08-31 Published:2019-08-31
  • About author:Xiang Chen received his B.S. degree in information management and information system from Xi'an Jiaotong University, Xi'an, in 2002. Then he received his M.S. and Ph.D. degrees in computer software and theory from Nanjing University, Nanjing, in 2008 and 2011 respectively. He is with the School of Information Science and Technology at Nantong University, Nantong, as an associate professor. His research interests are mainly in software maintenance and software testing, such as software defect prediction, security vulnerability prediction, combinatorial testing, regression testing, and software fault localization. He has published over 40 papers in referred journals or conferences, including Information and Software Technology, Journal of Systems and Software, IEEE Transactions on Reliability, Software Quality Journal, Journal of Computer Science and Technology, COMPSAC, APSEC and SAC, etc.
  • Supported by:
    This work was partially supported by the National Natural Science Foundation of China under Grant Nos. 61702041 and 61872263, the Open Project of State Key Laboratory for Novel Software Technology at Nanjing University under Grant No. KFKT2019B14, the Science and Technology Project of Beijing Municipal Education Commission under Grant No. KM201811232016, the Nantong Application Research Plan under Grant No. JC2018134, and Jiangsu Government Scholarship for Overseas Studies.

In current software defect prediction (SDP) research, most previous empirical studies only use datasets provided by PROMISE repository and this may cause a threat to the external validity of previous empirical results. Instead of SDP dataset sharing, SDP model sharing is a potential solution to alleviate this problem and can encourage researchers in the research community and practitioners in the industrial community to share more models. However, directly sharing models may result in privacy disclosure, such as model inversion attack. To the best of our knowledge, we are the first to apply differential privacy (DP) to privacy-preserving SDP model sharing and then propose a novel method DP-Share, since DP mechanisms can prevent this attack when the privacy budget is carefully selected. In particular, DP-Share first performs data preprocessing for the dataset, such as over-sampling for minority instances (i.e., defective modules) and conducting discretization for continuous features to optimize privacy budget allocation. Then, it uses a novel sampling strategy to create a set of training sets. Finally it constructs decision trees based on these training sets and these decision trees can form a random forest (i.e., model). The last phase of DP-Share uses Laplace and exponential mechanisms to satisfy the requirements of DP. In our empirical studies, we choose nine experimental subjects from real software projects. Then, we use AUC (area under ROC curve) as the performance measure and holdout as our model validation technique. After privacy and utility analysis, we find that DP-Share can achieve better performance than a baseline method DF-Enhance in most cases when using the same privacy budget. Moreover, we also provide guidelines to effectively use our proposed method. Our work attempts to fill the research gap in terms of differential privacy for SDP, which can encourage researchers and practitioners to share more SDP models and then effectively advance the state of the art of SDP.

Key words: software defect prediction; model sharing; differential privacy; cross project defect prediction; empirical study;

[1] Hall T, Beecham S, Bowes D, Gray D, Counsell S. A systematic literature review on fault prediction performance in software engineering. IEEE Transactions on Software Engineering, 2012, 38(6):1276-1304.
[2] Kamei Y, Shihab E. Defect prediction:Accomplishments and future challenges. In Proc. the 23rd International Conference on Software Analysis, Evolution, and Reengineering, March 2016, pp.33-45.
[3] Fredrikson M, Jha S, Ristenpart T. Model inversion attacks that exploit confidence information and basic countermeasures. In Proc. the 22nd ACM SIGSAC Conference on Computer and Communications Security, October 2015, pp.1322-1333.
[4] Hosseini S, Turhan B, Gunarathna D. A systematic literature review and meta-analysis on cross project defect prediction. IEEE Transactions on Software Engineering, 2019, 45(2):111-147.
[5] Dwork C. Differential privacy. In Proc. the 33rd International Colloquium on Automata, Languages and Programming, July 2006, pp.1-12.
[6] Zhu T, Li G, Zhou W, Yu P S. Differentially private data publishing and analysis:A survey. IEEE Transactions on Knowledge and Data Engineering, 2017, 29(8):1619-1638.
[7] Friedman A, Schuster A. Data mining with differential privacy. In Proc. the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 2010, pp.493-502.
[8] Chawla N V, Bowyer K W, Hall L O, Kegelmeyer W P. SMOTE:Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 2002, 16(1):321-357.
[9] Fayyad U. Multi-interval discretization of continuousvalued attributes for classification learning. In Proc. the 13th International Joint Conference on Artificial Intelligence, August 1993, pp.1022-1027.
[10] Patil A, Singh S. Differential private random forest. In Proc. the 2014 International Conference on Advances in Computing, Communications and Informatics, September 2014, pp.2623-2630.
[11] Zhang D, Chen X, Cui Z, Ju X. Software defect prediction model sharing under differential privacy. In Proc. the 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation, October 2018, pp.1547-1554.
[12] Tantithamthavorn C, Hassan A E. An experience report on defect modelling in practice:Pitfalls and challenges. In Proc. the 40th International Conference on Software Engineering:Software Engineering in Practice, May 2018, pp.286-295.
[13] Chen X, Zhao Y, Wang Q, Yuan Z. MULTI:Multi-objective effort-aware just-in-time software defect prediction. Information and Software Technology, 2018, 93:1-13.
[14] Radjenovic D, Hericko M, Torkar R, Zivkovic A. Software fault prediction metrics:A systematic literature review. Information and Software Technology, 2013, 55(8):1397-1418.
[15] Peters F, Menzies T. Privacy and utility for defect prediction:Experiments with MORPH. In Proc. the 34th International Conference on Software Engineering, June 2012, pp.189-199.
[16] Weyuker E J, Ostrand T J, Bell R M. Do too many cooks spoil the broth? Using the number of developers to enhance defect prediction models. Empirical Software Engineering, 2008, 13(5):539-559.
[17] Peters F, Menzies T, Gong L, Zhang H. Balancing privacy and utility in cross-company defect prediction. IEEE Transactions on Software Engineering, 2013, 39(8):1054-1068.
[18] Peters F, Menzies T, Layman L. LACE2:Better privacypreserving data sharing for cross project defect prediction. In Proc. the 37th IEEE/ACM International Conference on Software Engineering, May 2015, pp.801-811.
[19] Fan Y, Lv C, Zhang X, Zhou G, Zhou Y. The utility challenge of privacy-preserving data-sharing in cross-company defect prediction:An empirical study of the CLIFF & MORPH algorithm. In Proc. International Conference on Software Maintenance and Evolution, September 2017, pp.80-90.
[20] Blum A, Dwork C, McSherry F, Nissim K. Practical privacy:The SuLQ framework. In Proc. the 24th ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, June 2005, pp.128-138.
[21] Dwork C. Differential privacy:A survey of results. In Proc. the 5th International Conference on Theory and Applications of Models of Computation, April 2008, pp.1-19.
[22] Dwork C. A firm foundation for private data analysis. Communications of the ACM, 2011, 54(1):86-95.
[23] McSherry F, Talwar K. Mechanism design via differential privacy. In Proc. the 48th Annual IEEE Symposium on Foundations of Computer Science, October 2007, pp.94-103.
[24] McSherry F D. Privacy integrated queries:An extensible platform for privacy-preserving data analysis. In Proc. the 2009 ACM SIGMOD International Conference on Management of Data, June 2009, pp.19-30.
[25] Tan M, Tan L, Dara S, Mayeux C. Online defect prediction for imbalanced data. In Proc. the 37th IEEE/ACM International Conference on Software Engineering, May 2015, pp.99-108.
[26] Bennin K E, Keung J, Phannachitta P, Monden A, Mensah S. MAHAKIL:Diversity based oversampling approach to alleviate the class imbalance issue in software defect prediction. IEEE Transactions on Software Engineering, 2018, 44(6):534-550.
[27] Liu M, Miao L, Zhang D. Two-stage cost-sensitive learning for software defect prediction. IEEE Transactions on Reliability, 2014, 63(2):676-686.
[28] Wang S, Yao X. Using class imbalance learning for software defect prediction. IEEE Transactions on Reliability, 2013, 62(2):434-443.
[29] Öztürk M M. Which type of metrics are useful to deal with class imbalance in software defect prediction? Information and Software Technology, 2017, 92:17-29.
[30] He H, Garcia E A. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 2009, 21(9):1263-1284.
[31] García S, Luengo J, Sáez J A, López V, Herrera F. A survey of discretization techniques:Taxonomy and empirical analysis in supervised learning. IEEE Transactions on Knowledge and Data Engineering, 2013, 25(4):734-750.
[32] Hansen M H, Yu B. Model selection and the principle of minimum description length. Journal of the American Statistical Association, 2001, 96(454):746-774.
[33] Steinberg D. Cart:Classification and regression trees. In The Top Ten Algorithms in Data Mining, Wu X D, Kumer V (eds.), Chapman and Hall/CRC, 2009, pp.193-216.
[34] Wang S, Liu T, Tan L. Automatically learning semantic features for defect prediction. In Proc. the 38th International Conference on Software Engineering, May 2016, pp.297-308.
[35] Tantithamthavorn C, McIntosh S, Hassan A E, Matsumoto K. Automated parameter optimization of classification techniques for defect prediction models. In Proc. the 38th International Conference on Software Engineering, May 2016, pp.321-332.
[36] Zhang F, Zheng Q, Zou Y, Hassan A E. Cross-project defect prediction using a connectivity-based unsupervised classifier. In Proc. the 38th International Conference on Software Engineering, May 2016, pp.309-320.
[37] He P, Li B, Liu X, Chen J, Ma Y. An empirical study on software defect prediction with a simplified metric set. Information and Software Technology, 2015, 59:170-190.
[38] Sayyad Shirabad J, Menzies T J. The PROMISE repository of softare engineering databases. Technical Report, School of Information Technology and Engineering, University of Ottawa. http://promise.site.upttawa.ca/SERepsiting, Aug. 2018.
[39] Jureczko M, Madeyski L. Towards identifying software project clusters with regard to defect prediction. In Proc. the 6th International Conference on Predictive Models in Software Engineering, September 2010, Article No. 9.
[40] Chidamber S R, Kemerer C F. A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 1994, 20(6):476-493.
[41] Zhang Y, Lo D, Xia X, Sun J. An empirical study of classifier combination for cross-project defect prediction. In Proc. the 39th IEEE Annual Computer Software and Applications Conference, Volume 2, July 2015, pp.264-269.
[42] Liu W, Liu S, Gu Q, Chen J, Chen X, Chen D. Empirical studies of a two-stage data preprocessing approach for software fault prediction. IEEE Transactions on Reliability, 2016, 65(1):38-53.
[43] Liu S, Chen X, Liu W, Chen J, Gu Q, Chen D. FECAR:A feature selection framework for software defect prediction. In Proc. the 38th IEEE Annual Computer Software and Applications Conference, July 2014, pp.426-435.
[44] Tantithamthavorn C, McIntosh S, Hassan A E, Matsumoto K. An empirical comparison of model validation techniques for defect prediction models. IEEE Transactions on Software Engineering, 2017, 43(1):1-18.
[45] Dwork C, Feldman V, Hardt M, Pitassi T, Reingold O, Roth A. The reusable holdout:Preserving validity in adaptive data analysis. Science, 2015, 349(6248):636-638.
[46] Shivaji S, Whitehead E J, Akella R, Kim S. Reducing features to improve code change-based bug prediction. IEEE Transactions on Software Engineering, 2013, 39(4):552-569.
[47] Herbold S, Trautsch A, Grabowski J. A comparative study to benchmark cross-project defect prediction approaches. IEEE Transactions on Software Engineering, 2018, 44(9):811-833.
[48] Pan S J, Yang Q. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10):1345-1359.
[49] Wu F, Jing X Y, Sun Y, Sun J, Huang L, Cui F, Sun Y. Cross-project and within-project semisupervised software defect prediction:A unified approach. IEEE Transactions on Reliability, 2018, 67(2):581-597.
[50] Jing X Y, Wu F, Dong X, Xu B. An improved SDA based defect prediction framework for both within project and cross-project class-imbalance problems. IEEE Transactions on Software Engineering, 2017, 43(4):321-339.
[51] Ni C, Liu W S, Chen X, Gu Q, Chen D X, Huang Q G. A cluster based feature selection method for cross-project software defect prediction. Journal of Computer Science and Technology, 2017, 32(6):1090-1107.
[52] Krishna R, Menzies T, Fu W. Too much automation? The bellwether effect and its implications for transfer learning. In Proc. the 31st IEEE/ACM International Conference on Automated Software Engineering, August 2016, pp.122-131.
[53] Ryu D, Jang J I, Baik J. A hybrid instance selection using nearest-neighbor for cross-project defect prediction. Journal of Computer Science and Technology, 2015, 30(5):969-980.
[54] Hosseini S, Turhan B, Mantyla M. A benchmark study on the effectiveness of search-based data selection and feature selection for cross project defect prediction. Information and Software Technology, 2018, 95:296-312.
[55] Moser R, Pedrycz W, Succi G. A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. In Proc. the 30th International Conference on Software Engineering, May 2008, pp.181-190.
[56] Menzies T, Milton Z, Turhan B, Cukic B, Jiang Y, Bener A. Defect prediction from static code features:Current results, limitations, new approaches. Automated Software Engineering, 2010, 17(4):375-407.
[57] Storn R, Price K. Differential evolution-A simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization, 1997, 11(4):341-359.
[58] Agrawal A, Menzies T. Is "better data" better than "better data miners"?:On the benefits of tuning SMOTE for defect prediction. In Proc. the 40th International Conference on Software Engineering, May 2018, pp.1050-1061.
[59] Chen X, Zhang D, Zhao Y, Cui Z, Ni C. Software defect number prediction:Unsupervised vs supervised methods. Information and Software Technology, 2019, 106:161-181.
[1] William Croft, Jörg-Rüdiger Sack, and Wei Shi. Differential Privacy via a Truncated and Normalized Laplace Mechanism [J]. Journal of Computer Science and Technology, 2022, 37(2): 369-388.
[2] Jian-Zhe Zhao, Xing-Wei Wang, Ke-Ming Mao, Chen-Xi Huang, Yu-Kai Su, and Yu-Chen Li. Correlated Differential Privacy of Multiparty Data Release in Machine Learning [J]. Journal of Computer Science and Technology, 2022, 37(1): 231-251.
[3] Que-Ping Kong, Zi-Yan Wang, Yuan Huang, Xiang-Ping Chen, Xiao-Cong Zhou, Zi-Bin Zheng, and Gang Huang. Characterizing and Detecting Gas-Inefficient Patterns in Smart Contracts [J]. Journal of Computer Science and Technology, 2022, 37(1): 67-82.
[4] Yi-Xuan Tang, Zhi-Lei Ren, He Jiang, Xiao-Chen Li, Wei-Qiang Kong. An Empirical Comparison Between Tutorials and Crowd Documentation of Application Programming Interface [J]. Journal of Computer Science and Technology, 2021, 36(4): 856-876.
[5] Yong-Hao Wu, Zheng Li, Yong Liu, Xiang Chen. FATOC: Bug Isolation Based Multi-Fault Localization by Using OPTICS Clustering [J]. Journal of Computer Science and Technology, 2020, 35(5): 979-998.
[6] Mohammed Alqmase, Mohammad Alshayeb, Lahouari Ghouti. Threshold Extraction Framework for Software Metrics [J]. Journal of Computer Science and Technology, 2019, 34(5): 1063-1078.
[7] Chao Ni, Wang-Shu Liu, Xiang Chen, Qing Gu, Dao-Xu Chen, Qi-Guo Huang. A Cluster Based Feature Selection Method for Cross-Project Software Defect Prediction [J]. , 2017, 32(6): 1090-1107.
[8] Ning Wang, Yu Gu, Jia Xu, Fang-Fang Li, Ge Yu. Differentially Private Event Histogram Publication on Sequences over Graphs [J]. , 2017, 32(5): 1008-1024.
[9] Xin-Li Yang, David Lo, Xin Xia, Zhi-Yuan Wan, Jian-Ling Sun. What Security Questions Do Developers Ask? A Large-Scale Study of Stack Overflow Posts [J]. , 2016, 31(5): 910-924.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] Liu Mingye; Hong Enyu;. Some Covering Problems and Their Solutions in Automatic Logic Synthesis Systems[J]. , 1986, 1(2): 83 -92 .
[2] Chen Shihua;. On the Structure of (Weak) Inverses of an (Weakly) Invertible Finite Automaton[J]. , 1986, 1(3): 92 -100 .
[3] Gao Qingshi; Zhang Xiang; Yang Shufan; Chen Shuqing;. Vector Computer 757[J]. , 1986, 1(3): 1 -14 .
[4] Chen Zhaoxiong; Gao Qingshi;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[5] Huang Heyan;. A Parallel Implementation Model of HPARLOG[J]. , 1986, 1(4): 27 -38 .
[6] Min Yinghua; Han Zhide;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[7] Tang Tonggao; Zhao Zhaokeng;. Stack Method in Program Semantics[J]. , 1987, 2(1): 51 -63 .
[8] Min Yinghua;. Easy Test Generation PLAs[J]. , 1987, 2(1): 72 -80 .
[9] Zhu Hong;. Some Mathematical Properties of the Functional Programming Language FP[J]. , 1987, 2(3): 202 -216 .
[10] Li Minghui;. CAD System of Microprogrammed Digital Systems[J]. , 1987, 2(3): 226 -235 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

Home
Editorial Board
Author Guidelines
Subscription
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved