We use cookies to improve your experience with our site.
Dong Yan, Xu-Sen Yin, Cheng Lian, Xiang Zhong, Xin Zhou, Gan-Sha Wu. Using Memory in the Right Way to Accelerate Big Data Processing[J]. Journal of Computer Science and Technology, 2015, 30(1): 30-41. DOI: 10.1007/s11390-015-1502-9
Citation: Dong Yan, Xu-Sen Yin, Cheng Lian, Xiang Zhong, Xin Zhou, Gan-Sha Wu. Using Memory in the Right Way to Accelerate Big Data Processing[J]. Journal of Computer Science and Technology, 2015, 30(1): 30-41. DOI: 10.1007/s11390-015-1502-9

Using Memory in the Right Way to Accelerate Big Data Processing

  • Big data processing is becoming a standout part of data center computation. However, latest research has indicated that big data workloads cannot make full use of modern memory systems. We find that the dramatic inefficiency of the big data processing is from the enormous amount of cache misses and stalls of the depended memory accesses. In this paper, we introduce two optimizations to tackle these problems. The first one is the slice-and-merge strategy, which reduces the cache miss rate of the sort procedure. The second optimization is direct-memory-access, which reforms the data structure used in key/value storage. These optimizations are evaluated with both micro-benchmarks and the real-world benchmark HiBench. The results of our micro-benchmarks clearly demonstrate the effectiveness of our optimizations in terms of hardware event counts; and the additional results of HiBench show the 1.21X average speedup on the application-level. Both results illustrate that careful hardware/software co-design will improve the memory efficiency of big data processing. Our work has already been integrated into Intel distribution for Apache Hadoop.
  • loading

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return