We use cookies to improve your experience with our site.

CirroData:另一个基于SQL-on-Hadoop的高性能数据分析引擎

CirroData: Yet Another SQL-on-Hadoop Data Analytics Engine with High Performance

  • 摘要: 本文介绍了一个针对大数据分析工作负载而设计的高性能SQL-on-Hadoop数据库系统-CirroData。这个自研的面向企业级联机分析处理的大规模并行数据库系统已有超过七年的历史。本文主要介绍CirroData如何实现高性能数据处理的设计与实现细节,其中包含的多种优化技术在文中都有讨论。CirroData客户的日常使用证明了这些技术的有效性和高性能。通过基于TPC-H的标准化测试和多个实际应用案例的评测及检验表明,CirroData的性能优于社区中同类的多种数据库系统,例如Spark+Hive、Spark+HBase、Impala、DB-X/Y、Greenplum和HAWQ等。在TPC-H的查询负载下,CirroData与Greenplum、HAWQ和Spark的性能加速比可达4.99倍。实际应用评估结果表明,CirroData的性能明显优于Spark+Hive、Spark+HBase,性能提升可以分别达到8.4倍和38.8倍。与此同时,与Greenplum、DB-X、Impala、DB-Y和HAWQ相比,CirroData在有些应用负载上可以获得的性能加速比分别可达20倍、100倍、182.5倍、92.6倍和55.5倍。

     

    Abstract: This paper presents CirroData, a high-performance SQL-on-Hadoop system designed for Big Data analytics workloads. As a home-grown enterprise-level online analytical processing (OLAP) system with more than seven-year research and development (R&D) experiences, we share our design details to the community about how to achieve high performance in CirroData. Multiple optimization techniques have been discussed in the paper. The effectiveness and the efficiency of all these techniques have been proved by our customers' daily usage. Benchmark-level studies, as well as several real application case studies of CirroData, have been presented in this paper. Our evaluations show that CirroData can outperform various types of counterpart database systems in the community, such as "Spark+Hive", "Spark+HBase", Impala, DB-X/Y, Greenplum, HAWQ, and others. CirroData can achieve up to 4.99x speedup compared with Greenplum, HAWQ, and Spark in the standard TPC-H queries. Application-level evaluations demonstrate that CirroData outperforms "Spark+Hive" and "Spark+HBase" by up to 8.4x and 38.8x, respectively. In the meantime, CirroData achieves the performance speedups for some application workloads by up to 20x, 100x, 182.5x, 92.6x, and 55.5x as compared with Greenplum, DB-X, Impala, DB-Y, and HAWQ, respectively.

     

/

返回文章
返回