We use cookies to improve your experience with our site.

Indexed in:

SCIE, EI, Scopus, INSPEC, DBLP, CSCD, etc.

Submission System
(Author / Reviewer / Editor)
Advanced Search
Volume 34 Issue 6
November  2019
Turn off MathJax
Article Contents
Abstract
References
Related Articles
Supplements
Rui Ren, Jiechao Cheng, Xi-Wen He, Lei Wang, Jian-Feng Zhan, Wan-Ling Gao, Chun-Jie Luo. HybridTune: Spatio-Temporal Performance Data Correlation for Performance Diagnosis of Big Data Systems[J]. Journal of Computer Science and Technology, 2019, 34(6): 1167-1184. DOI: 10.1007/s11390-019-1968-y
Citation: Rui Ren, Jiechao Cheng, Xi-Wen He, Lei Wang, Jian-Feng Zhan, Wan-Ling Gao, Chun-Jie Luo. HybridTune: Spatio-Temporal Performance Data Correlation for Performance Diagnosis of Big Data Systems[J]. Journal of Computer Science and Technology, 2019, 34(6): 1167-1184. DOI: 10.1007/s11390-019-1968-y
shu

HybridTune: Spatio-Temporal Performance Data Correlation for Performance Diagnosis of Big Data Systems

  • 1 Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China;
    2 University of Chinese Academy of Sciences, Beijing 100049, China;
    3 School of Computing, National University of Singapore, Singapore 117417, Singapore

Funds: This work is supported by the National Key Research and Development Program of China under Grant No. 2016YFB1000601.
More Information
  • Author Bio:

    Rui Ren received her B.S. degree in computer science from the Sichuan University, Chengdu, in 2009, her M.S. degree in computer architecture from Chinese Academy of Sciences, Beijing, in 2012, and her Ph.D. degree in computer software and theory from Chinese Academy of Sciences, Beijing, in 2019. She is currently an engineer in the Institute of Computing Technology, Chinese Academy of Sciences, Beijing. Her research interests include big data, performance analysis and optimization.

  • Corresponding author:

    Jian-Feng Zhan E-mail: zhanjianfeng@ict.ac.cn

  • Received Date: September 05, 2018
  • Revised Date: September 03, 2019
  • Published Date: November 15, 2019
    Abstract
  • With tremendous growing interests in Big Data, the performance improvement of Big Data systems becomes more and more important. Among many steps, the first one is to analyze and diagnose performance bottlenecks of the Big Data systems. Currently, there are two major solutions. One is the pure data-driven diagnosis approach, which may be very time-consuming; the other is the rule-based analysis method, which usually requires prior knowledge. For Big Data applications like Spark workloads, we observe that the tasks in the same stages normally execute the same or similar codes on each data partition. On basis of the stage similarity and distributed characteristics of Big Data systems, we analyze the behaviors of the Big Data applications in terms of both system and micro-architectural metrics of each stage. Furthermore, for different performance problems, we propose a hybrid approach that combines prior rules and machine learning algorithms to detect performance anomalies, such as straggler tasks, task assignment imbalance, data skew, abnormal nodes and outlier metrics. Following this methodology, we design and implement a lightweight, extensible tool, named HybridTune, and measure the overhead and anomaly detection effectiveness of HybridTune using the BigDataBench benchmarks. Our experiments show that the overhead of HybridTune is only 5%, and the accuracy of outlier detection algorithm reaches up to 93%. Finally, we report several use cases diagnosing Spark and Hadoop workloads using BigDataBench, which demonstrates the potential use of HybridTune.
    • FullText(HTML)
    • References (33)
  • [1]
    Dai J, Huang J, Huang S, Huang B, Liu Y. HiTune:Dataflow-based performance analysis for big data cloud. In Proc. the 2011 USENIX Conference on USENIX Annual Technical Conference, June 2011, Article No. 27.
    [2]
    Guo Q, Li Y, Liu T, Wang K, Chen G, Bao X, Tang W. Correlation-based performance analysis for full-system MapReduce optimization. In Proc. the 2013 IEEE International Conference on Big Data, October 2013, pp.753-761.
    [3]
    Garduño E, Kavulya S P, Tan J, Gandhi R, Narasimhan P. Theia:Visual signatures for problem diagnosis in large Hadoop clusters. In Proc. the 26th Large Installation System Administration Conference, December 2012, pp.33-42.
    [4]
    Tan J, Pan X, Kavulya S, Gandhi R, Narasimhan P. Mochi:Visual log-analysis based tools for debugging Hadoop. In Proc. USENIX Workshop on Hot Topics in Cloud Computing, June 2009, Article No. 1.
    [5]
    Cretu-Ciocarlie G, Budiu M, Goldszmidt M. Hunting for problems with Artemis. In Proc. the 1st USENIX Workshop on Analysis of System Logs, Dec. 2008, Article No. 2.
    [6]
    Herodotou H, Lim H, Luo G, Borisov N, Dong L, Cetin F, Babu S. Starfish:A self-tuning system for big data analytics. In Proc. the 5th Biennial Conference on Innovative Data Systems Research, January 2011, pp.261-272.
    [7]
    Wang L, Zhan J, Luo C, Zhu Y, Yang Q, He Y, Gao W, Jia Z, Shi Y, Zhang S, Zheng C, Lu G, Zhan K, Qiu B. BigDataBench:A Big Data benchmark suite from internet services. In Proc. the 20th IEEE International Symposium on High Performance Computer Architecture, February 2014, pp.488-499.
    [8]
    Ananthanarayanan G, Kandula S, Greenberg A, Stoica I, Lu Y, Saha B, Harris E. Reining in the outliers in MapReduce clusters using Mantri. In Proc. the 9th USENIX Conference on Operating Systems Design and Implementation, October 2010, pp.265-278.
    [9]
    Zaharia M, Chowdhury M, Das T, Dave A, Ma J, McCauley M, Franklin M, Shenker S, Stoica I. Resilient distributed datasets:A fault-tolerant abstraction for in-memory cluster computing. In Proc. the 9th USENIX Symposium on Networked Systems Design and Implementation, April 2012, pp.15-28.
    [10]
    Isard M, Budiu M, Yu Y, Birrell A, Fetterly D. Dryad:Distributed data-parallel programs from sequential building blocks. In Proc. the 2007 EuroSys Conference, March 2007, pp.59-72.
    [11]
    Ren R, Jia Z, Wang L, Zhan J, Yi T. BDTUne:Hierarchical correlation-based performance analysis and rule-based diagnosis for big data systems. In Proc. the IEEE International Conference on Big Data, Dec. 2016, pp.555-562.
    [12]
    Cochran W, Cooley J, Favin D, Helms H, Kaenel R, Langa W, Maling G, Nelson D, Rader C, Welch P. What is the fast Fourier transform? IEEE Transactions on Audio and Electroacoustics, 1967, 55(10):1664-1674.
    [13]
    Knorr E M, Ng R T. Algorithms for mining distancebased outliers in large datasets. In Proc. the 24th International Conference on Very Large Data Bases, August 1998, pp.392-403.
    [14]
    Ming Z, Luo C, Gao W, Han R, Yang Q, Wang L, Zhan J. BDGS:A scalable Big Data generator suite in Big Data benchmarking. In Proc. the 2013 Workshop Series on Big Data Benchmarking, July 2014, pp.138-154.
    [15]
    Meng X, Bradley J, Yavuz B, Sparks E, Venkataraman S, Liu D, Freeman J, Tsai D B, Amde M, Owen S, Xin D, Xin R, Franklin M J, Zadeh R, Zaharia M, Talwalkar A. MLlib:Machine learning in Apache Spark. J. Mach. Learn. Res., 2016, 17:Article No. 34.
    [16]
    Wang C, Talwar V, Schwan K, Ranganathan P. Online detection of utility cloud anomalies using metric distributions. In Proc. the IEEE/IFIP Network Operations and Management Symposium, April 2010, pp.96-103.
    [17]
    Ousterhout K, Rasti R, Ratnasamy S, Shenker S, Chun B. Making sense of performance in data analytics frameworks. In Proc. the 12th USENIX Symposium on Networked Systems Design and Implementation, May 2015, pp.293-307.
    [18]
    Jayathilaka H, Krintz C, Wolski R. Detecting performance anomalies in cloud platform applications. IEEE Transactions on Cloud Computing. doi: 10.1109/TCC.2018.2808289.
    [19]
    Ramaswamy S, Rastogi R, Shim K. Efficient algorithms for mining outliers from large data sets. In Proc. the 2000 ACM SIGMOD International Conference on Management of Data, May 2000, pp.427-438.
    [20]
    Breunig M M, Kriegel H P, Ng R T, Sander J. LOF:Identifying density-based local outliers. In Proc. ACM SIGMOD International Conference on Management of Data, May 2000, pp.93-104.
    [21]
    Yu D, Sheikholeslami G, Zhang A. FindOut:Finding outliers in very large datasets. Knowledge and Information Systems, 2002, 4(4):387-412.
    [22]
    Yu L, Lan Z. A scalable, non-parametric method for detecting performance anomaly in large scale computing. IEEE Transactions on Parallel and Distributed Systems, 2016, 27(7):1902-1914.
    [23]
    Tan J, Pan X, Marinelli E, Kavulya S, Gandhi R, Narasimhan P. Kahuna:Problem diagnosis for MapReducebased cloud computing environments. In Proc. the IEEE/IFIP Network Operations and Management Symposium, April 2010, pp.112-119.
    [24]
    Pan X, Tan J, Kavulya S, Gandhi R, Narasimhan P. Ganesha:BlackBox diagnosis of MapReduce systems. SIGMETRICS Performance Evaluation Review, 2009, 37(3):8-13.
    [25]
    Gupta C, Sinha R, Zhang Y. Eagle:User profile-based anomaly detection for securing Hadoop clusters. In Proc. the 2015 IEEE International Conference on Big Data, October 2015, pp.1336-1343.
    [26]
    Kasick M P, Tan J, Gandhi R, Narasimhan P. Black-box problem diagnosis in parallel file systems. In Proc. the 8th USENIX Conference on File and Storage Technologies, February 2010, pp.43-56.
    [27]
    Fu X, Ren R, McKeez S A, Zhan J, Sun N. Digging deeper into cluster system logs for failure prediction and root cause diagnosis. In Proc. IEEE International Conference on Cluster Computing, September 2014, pp.103-112.
    [28]
    Khan L, Awad M, Thuraisingham B. A new intrusion detection system using support vector machines and hierarchical clustering. The VLDB Journal, 2007, 16(4):507-521.
    [29]
    Lee S, Shin K G. Probabilistic diagnosis of multiprocessor systems. ACM Computing Surveys, 1994, 26(1):121-139.
    [30]
    Das K, Schneider J. Detecting anomalous records in categorical datasets. In Proc. the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2007, pp.220-229.
    [31]
    Mi H, Wang H, Zhou Y, Lyu M R, Cai H. Toward finegrained, unsupervised, scalable performance diagnosis for production cloud computing systems. IEEE Transactions on Parallel and Distributed Systems, 2013, 24(6):1245-1255.
    [32]
    Jia T, Chen P, Yang L, Li Y, Meng F, Xu J. An approach for anomaly diagnosis based on hybrid graph model with logs for distributed services. In Proc. the 2017 IEEE International Conference on Web Services, June 2017, pp.25-32.
    [33]
    Ren R, Tian S, Wang L. Online anomaly detection framework for Spark systems via stage-task behavior modeling. In Proc. the 15th ACM International Conference on Computing Frontiers, May 2018, pp.256-259.
    • Relative Articles

  • Related Articles

    [1]Hui-Jing Yang, Juan Fang, Min Cai, Zhi Cai. A Prefetch-Adaptive Intelligent Cache Replacement Policy Based on Machine Learning[J]. Journal of Computer Science and Technology, 2023, 38(2): 391-404. DOI: 10.1007/s11390-022-1573-3
    [2]Jian-Zhe Zhao, Xing-Wei Wang, Ke-Ming Mao, Chen-Xi Huang, Yu-Kai Su, Yu-Chen Li. Correlated Differential Privacy of Multiparty Data Release in Machine Learning[J]. Journal of Computer Science and Technology, 2022, 37(1): 231-251. DOI: 10.1007/s11390-021-1754-5
    [3]Yi Zhong, Jian-Hua Feng, Xiao-Xin Cui, Xiao-Le Cui. Machine Learning Aided Key-Guessing Attack Paradigm Against Logic Block Encryption[J]. Journal of Computer Science and Technology, 2021, 36(5): 1102-1117. DOI: 10.1007/s11390-021-0846-6
    [4]Andrea Caroppo, Alessandro Leone, Pietro Siciliano. Comparison Between Deep Learning Models and Traditional Machine Learning Approaches for Facial Expression Recognition in Ageing Adults[J]. Journal of Computer Science and Technology, 2020, 35(5): 1127-1146. DOI: 10.1007/s11390-020-9665-4
    [5]Shu-Zheng Zhang, Zhen-Yu Zhao, Chao-Chao Feng, Lei Wang. A Machine Learning Framework with Feature Selection for Floorplan Acceleration in IC Physical Design[J]. Journal of Computer Science and Technology, 2020, 35(2): 468-474. DOI: 10.1007/s11390-020-9688-x
    [6]Xiao-Chen Li, Wen-Ji Mao, Daniel Zeng, Peng Su, Fei-Yue Wang. Performance Evaluation of Machine Learning Methods in Cultural Modeling[J]. Journal of Computer Science and Technology, 2009, 24(6): 1010-1017.
    [7]Zhi-Hua Zhou. Multi-Instance Learning from Supervised View[J]. Journal of Computer Science and Technology, 2006, 21(5): 800-809.
    [8]Xueqi Cheng, Songbo Tan, Lilian Tang. Using DragPushing to Refine Concept Index for Text Categorization[J]. Journal of Computer Science and Technology, 2006, 21(4): 592-596.
    [9]Wang Shitong, E.Scottand, A.Gamermann. Extract Rules by Using Rough Set and Knowledge-Based NN[J]. Journal of Computer Science and Technology, 1998, 13(3): 279-284.
    [10]Wu Xindong. Inductive Learning[J]. Journal of Computer Science and Technology, 1993, 8(2): 22-36.
    • Supplements (2)

  • Others

  • Cited by

    Periodical cited type(2)

    1. Rui Ren, JieChao Cheng, Hao Shi, et al. Intelligent Computing and Block Chain. Communications in Computer and Information Science, DOI:10.1007/978-981-16-1160-5_11
    2. Wenbin Wang. 2021 International Conference on Applications and Techniques in Cyber Intelligence. Lecture Notes on Data Engineering and Communications Technologies, DOI:10.1007/978-3-030-79197-1_126

    Other cited types(0)

Catalog

    Get Citation
    PDF
    XML
    Article views (89) PDF downloads (19) Cited by(2)
    Related

    Copyright ©2025 JCST, All Rights Reserved     京ICP备05002829号-1

    Institute of Computing Technology, Chinese Academy of Sciences
    P.O. Box 2704, Beijing 100190, P.R. China

    Tel.: 86-10-62610746 E-mail: jcst@ict.ac.cn

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return