We use cookies to improve your experience with our site.

Indexed in:

SCIE, EI, Scopus, INSPEC, DBLP, CSCD, etc.

Submission System
(Author / Reviewer / Editor)
Advanced Search
Volume 34 Issue 1
January  2019
Turn off MathJax
Article Contents
Abstract
References
Supplements
Yang Hong, Yang Zheng, Fan Yang, Bin-Yu Zang, Hai-Bing Guan, Hai-Bo Chen. Scaling out NUMA-Aware Applications with RDMA-Based Distributed Shared Memory[J]. Journal of Computer Science and Technology, 2019, 34(1): 94-112. DOI: 10.1007/s11390-019-1901-4
Citation: Yang Hong, Yang Zheng, Fan Yang, Bin-Yu Zang, Hai-Bing Guan, Hai-Bo Chen. Scaling out NUMA-Aware Applications with RDMA-Based Distributed Shared Memory[J]. Journal of Computer Science and Technology, 2019, 34(1): 94-112. DOI: 10.1007/s11390-019-1901-4
shu

Scaling out NUMA-Aware Applications with RDMA-Based Distributed Shared Memory

  • Shanghai Key Laboratory for Scalable Computing Systems, Shanghai Jiao Tong University, Shanghai 200240, China

Funds: This work was supported in part by the National Key Research and Development Program of China under Grant No. 2016YFB1000500, the National Natural Science Foundation of China under Grant No. 61572314, and the National Youth Top-Notch Talent Support Program of China.
More Information
  • Author Bio:

    Yang Hong is currently a Ph.D. candidate at Shanghai Key Laboratory for Scalable Computing Systems, Shanghai Jiao Tong University, Shanghai. He received his B.S. degree in software engineering from Shanghai Jiao Tong University, Shanghai, in 2013. His research interests include computer architecture, virtualization, parallel computing, and networked systems.

  • Corresponding author:

    Hai-Bo Chen E-mail: haibochen@sjtu.edu.cn

  • Received Date: June 12, 2018
  • Revised Date: November 20, 2018
  • Published Date: January 04, 2019
    Abstract
  • The multicore evolution has stimulated renewed interests in scaling up applications on shared-memory multiprocessors, significantly improving the scalability of many applications. But the scalability is limited within a single node; therefore programmers still have to redesign applications to scale out over multiple nodes. This paper revisits the design and implementation of distributed shared memory (DSM) as a way to scale out applications optimized for non-uniform memory access (NUMA) architecture over a well-connected cluster. This paper presents MAGI, an efficient DSM system that provides a transparent shared address space with scalable performance on a cluster with fast network interfaces. MAGI is unique in that it presents a NUMA abstraction to fully harness the multicore resources in each node through hierarchical synchronization and memory management. MAGI also exploits the memory access patterns of big-data applications and leverages a set of optimizations for remote direct memory access (RDMA) to reduce the number of page faults and the cost of the coherence protocol. MAGI has been implemented as a user-space library with pthread-compatible interfaces and can run existing multithreaded applications with minimized modifications. We deployed MAGI over an 8-node RDMAenabled cluster. Experimental evaluation shows that MAGI achieves up to 9.25x speedup compared with an unoptimized implementation, leading to a scalable performance for large-scale data-intensive applications.
    • FullText(HTML)
    • References (51)
  • [1]
    Dice D, Marathe V J, Shavit N. Lock cohorting:A general technique for designing NUMA locks. In Proc. the 17th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2012, pp.247-256.
    [2]
    Calciu I, Dice D, Lev Y, Luchangco V, Marathe V J, Shavit N. Numa-aware reader-writer locks. In Proc. the 18th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2013, pp.157-166.
    [3]
    Boyd-Wickizer S, Kaashoek M F, Morris R, Zeldovich N. OpLog:A library for scaling update-heavy data structures. Technical Report, Massachusetts Institute of Technology, 2014. https://dspace.mit.edu/handle/1721.1/89653, September 2018.
    [4]
    Majo Z, Gross T R. (Mis)understanding the NUMA memory system performance of multithreaded workloads. In Proc. the 2013 IEEE International Symposium on Workload Characterization, September 2013, pp.11-22.
    [5]
    Majo Z, Gross T. R. A library for portable and composable data locality optimizations for NUMA systems. In Proc. the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2015, pp.227-238.
    [6]
    Zhang K, Chen R, Chen H. NUMA-aware graph-structured analytics. In Proc. the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2015, pp.183-193.
    [7]
    Calciu I, Sen S, Balakrishnan M, Aguilera M K. Blackbox concurrent data structures for NUMA architectures. In Proc. the 22nd International Conference on Architectural Support for Programming Languages and Operating Systems, April 2017, pp.207-221.
    [8]
    Li K, Hudak P. Memory coherence in shared virtual memory systems. ACM Transactions on Computer Systems, 1989, 7(4):321-359.
    [9]
    Bennett J K, Carter J B, Zwaenepoel W. Munin:Distributed shared memory based on type-specific memory coherence. In Proc. the 2nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, March 1990, pp.168-176.
    [10]
    Chapman M, Heiser G. vNUMA:A virtual shared-memory multiprocessor. In Proc. the 2009 USENIX Annual Technical Conference, June 2009, pp.349-362.
    [11]
    Keleher P, Cox A L, Dwarkadas S, Zwaenepoel W. TreadMarks:Distributed shared memory on standard workstations and operating systems. In Proc. the USENIX Winter 1994 Technical Conference, January 1994, pp.115-132.
    [12]
    Fleisch B, Popek G. Mirage:A coherent distributed shared memory design. In Proc. the 12th ACM Symposium on Operating Systems Principles, December 1989, pp.211-223.
    [13]
    Iftode L, Singh J P, Li K. Scope consistency:A bridge between release consistency and entry consistency. In Proc. the 8th Annual ACM Symposium on Parallel Algorithms and Architectures, June 1996, pp.277-287.
    [14]
    Bershad B N, Zekauskas M J, Sawdon W A. The Midway distributed shared memory system. In Proc. Digest of Papers. Compcon Spring, February 1993, pp.528-537.
    [15]
    Erlichson A, Nuckolls N, Chesson G, Hennessy J. SoftFLASH:Analyzing the performance of clustered distributed virtual shared memory. In Proc. the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, October 1996, pp.210-220.
    [16]
    Stets R, Dwarkadas S, Hardavellas N, Hunt G, Kontothanassis L, Parthasarathy S, Scott M. Cashmere-2L:Software coherent shared memory on a clustered remotewrite network. In Proc. the 16th ACM Symposium on Operating Systems Principles, October 1997, pp.170-183.
    [17]
    Charles P, Grothof C, Saraswat V, Donawa C, Kielstra A, Ebcioglu K, von Praun C, Sarkar V. X10:An objectoriented approach to non-uniform cluster computing. In Proc. the 20th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, October 2005, pp.519-538.
    [18]
    Dragojević A, Narayanan D, Hodson O, Castro M. FaRM:Fast remote memory. In Proc. the 11th USENIX Symposium on Networked Systems Design and Implementation, April 2014, pp.401-414.
    [19]
    Wei X, Shi J, Chen Y, Chen R, Chen H. Fast in-memory transaction processing using RDMA and HTM. In Proc. the 25th Symposium on Operating Systems Principles, October 2015, pp.87-104.
    [20]
    Nelson J, Holt B, Myers B, Briggs P, Ceze L, Kahan S, Oskin M. Latency-tolerant software distributed shared memory. In Proc. the 2015 USENIX Annual Technical Conference, July 2015, pp.291-305.
    [21]
    Kalia A, Kaminsky M, Andersen D G. Design guidelines for high performance RDMA systems. In Proc. the 2016 USENIX Annual Technical Conference, June 2016, pp.437-450.
    [22]
    Kumar M, Maass S, Kashyap S, Veselý J, Yan Z, Kim T, Bhattacharjee A, Krishna T. LATR:Lazy translation coherence. In Proc. the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems, March 2018, pp.651-664.
    [23]
    Dennard R H, Gaensslen F H, Rideout V L, Bassous E, LeBlanc A R. Design of ion-implanted MOSFET's with very small physical dimensions. IEEE Journal of SolidState Circuits, 1974, 9(5):256-268.
    [24]
    Esmaeilzadeh H, Blem E, Amant R S, Sankaralingam K, Burger D. Dark silicon and the end of multicore scaling. In Proc. the 38th International Symposium on Computer Architecture, June 2011, pp.365-376.
    [25]
    Borkar S. Thousand core chips:A technology perspective. In Proc. the 44th Annual Design Automation Conference, June 2007, pp.746-749.
    [26]
    Ranger C, Raghuraman R, Penmetsa A, Bradski G, Kozyrakis C. Evaluating MapReduce for multi-core and multiprocessor systems. In Proc. the 13th International Symposium on High-Performance Computer Architecture, February 2007, pp.13-24.
    [27]
    Chen R, Chen H, Zang B. Tiled-MapReduce:Optimizing resource usages of data-parallel applications on multicore with tiling. In Proc. the 19th International Conference on Parallel Architectures and Compilation Techniques, September 2010, pp.523-534.
    [28]
    Zhu X, Chen W, Zheng W, Ma X. Gemini:A computationcentric distributed graph processing system. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, November 2016, pp.301-316.
    [29]
    Guo C, Wu H, Deng Z, Soni G, Ye J, Padhye J, Lipshteyn M. RDMA over commodity Ethernet at scale. In Proc. the 2016 ACM SIGCOMM Conference, August 2016, pp.202-215.
    [30]
    Scales D J, Gharachorloo K, Thekkath C A. Shasta:A low overhead, software-only approach for supporting fine-grain shared memory. In Proc. the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, October 1996, pp.174-185.
    [31]
    Karlsson M, Stenström P. Performance evaluation of a cluster-based multiprocessor built from ATM switches and bus-based multiprocessor servers. In Proc. the 2nd International Symposium on High-Performance Computer Architecture, February 1996, pp.4-13.
    [32]
    Gaud F, Lepers B, Decouchant J, Funston J, Fedorova A, Quéma V. Large pages may be harmful on NUMA systems. In Proc. the 2014 USENIX Annual Technical Conference, June 2014, pp.231-242.
    [33]
    Tsai S, Zhang Y. LITE kernel RDMA support for datacenter applications. In Proc. the 26th Symposium on Operating Systems Principles, October 2017, pp.306-324.
    [34]
    Bienia C, Kumar S, Singh J P, Li K. The PARSEC benchmark suite:Characterization and architectural implications. In Proc. the 17th International Conference on Parallel Architectures and Compilation Techniques, October 2008, pp.72-81.
    [35]
    Shun J, Blelloch G E. Ligra:A lightweight graph processing framework for shared memory. In Proc. the 18th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2013, pp.135-146.
    [36]
    Boldi P, Vigna S. The WebGraph framework I:Compression techniques. In Proc. the 13th International Conference on World Wide Web, May 2004, pp.595-602.
    [37]
    Boldi P, Rosa M, Santini M, Vigna S. Layered label propagation:A multiresolution coordinate-free ordering for compressing social networks. In Proc. the 20th International Conference on World Wide Web, March 2011, pp.587-596.
    [38]
    Raikin S, Liss L, Shachar A, Bloch N, Kagan M. Remote transactional memory. US Patent US20150269116, 2015. http://www.freepatentsonline.com/20150269116.pdf, September 2018.
    [39]
    Daglis A, Ustiugov D, Novaković S, Bugnion E, Falsafi B, Grot B. SABRes:Atomic object reads for in-memory rackscale computing. In Proc. the 49th Annual ACM/IEEE International Symposium on Microarchitecture, October 2016, Article No. 6.
    [40]
    Blumrich M A, Li K, Alpert R, Dubnicki C, Felten E W, Sandberg J. Virtual memory mapped network interface for the SHRIMP multicomputer. In Proc. the 21st Annual International Symposium on Computer Architecture, April 1994, pp.142-153.
    [41]
    Kontothanassis L I, Scott M L. Using memory-mapped network interfaces to improve the performance of distributed shared memory. In Proc. the 2nd International Symposium on High-Performance Computer Architecture, February 1996, pp.166-177.
    [42]
    Kalia A, Kaminsky M, Andersen D G. Using RDMA efficiently for key-value services. In Proc. the 2014 ACM Conference on SIGCOMM, August 2014, pp.295-306.
    [43]
    Dragojević A, Narayanan D, Nightingale E B, Renzelmann M, Shamis A, Badam A, Castro M. No compromises:Distributed transactions with consistency, availability, and performance. In Proc. the 25th ACM Symposium on Operating Systems Principles, October 2015, pp.54-70.
    [44]
    Kalia A, Kaminsky M, Andersen D G. FaSST:Fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, November 2016, pp.185-201.
    [45]
    Vasilevsky A, Lively D, Ofsthun S. Linuxr virtualization on Virtual IronTM VFe. In Proc. the 2005 Ottawa Linux Symposium, July 2005, pp.235-250.
    [46]
    Kaneda K, Oyama Y, Yonezawa A. A virtual machine monitor for providing a single system image. In Proc. the 17th IPSJ Computer System Symposium, November 2005, pp.3-12.
    [47]
    Gillett R B. Memory channel network for PCI. IEEE Micro, 1996, 16(1):12-18.
    [48]
    Blumrich M A, Alpert R D, Chen Y et al. Design choices in the SHRIMP system:An empirical study. In Proc. the 25th Annual International Symposium on Computer Architecture, June 1998, pp.330-341.
    [49]
    Zhou Y, Iftode L, Li K. Performance evaluation of two home-based lazy release consistency protocols for shared virtual memory systems. In Proc. the 2nd USENIX Symposium on Operating Systems Design and Implementation, October 1996, pp.75-88.
    [50]
    Yeung D, Kubiatowicz J, Agarwal A. MGS:A multigrain shared memory system. In Proc. the 23rd Annual International Symposium on Computer Architecture, May 1996, pp.44-55.
    [51]
    Novakovic S, Daglis A, Bugnion E, Falsafi B, Grot B. Scaleout NUMA. In Proc. the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, March 2014, pp.3-18.
    • Relative Articles
  • Supplements (1)

  • Others

  • Cited by

    Periodical cited type(8)

    1. Xingda Wei, Fangming Lu, Zhuobin Huang, et al. Towards Serialization/Deserialization-free State Transfer in Serverless Workflows. ACM Transactions on Computer Systems, 2025. DOI:10.1145/3725986
    2. Mohammad Ewais, Paul Chow. Disaggregated Memory in the Datacenter: A Survey. IEEE Access, 2023, 11: 20688. DOI:10.1109/ACCESS.2023.3250407
    3. Xingguo Jia, Jin Zhang, Boshi Yu, et al. GiantVM: A Novel Distributed Hypervisor for Resource Aggregation with DSM-aware Optimizations. ACM Transactions on Architecture and Code Optimization, 2022, 19(2): 1. DOI:10.1145/3505251
    4. Ting Wu, Xianzhang Chen, Kai Liu, et al. HydraFS: an efficient NUMA-aware in-memory file system. Cluster Computing, 2020, 23(2): 705. DOI:10.1007/s10586-019-02952-y
    5. Fangming Lu, Xingda Wei, Zhuobin Huang, et al. Serialization/Deserialization-free State Transfer in Serverless Workflows. Proceedings of the Nineteenth European Conference on Computer Systems, DOI:10.1145/3627703.3629568
    6. Jacob Nelson, Roberto Palmieri. Performance Evaluation of the Impact of NUMA on One-sided RDMA Interactions. 2020 International Symposium on Reliable Distributed Systems (SRDS), DOI:10.1109/SRDS51746.2020.00036
    7. Wataru Endo, Shigeyuki Sato, Kenjiro Taura. MENPS: A Decentralized Distributed Shared Memory Exploiting RDMA. 2020 IEEE/ACM Fourth Annual Workshop on Emerging Parallel and Distributed Runtime Systems and Middleware (IPDRM), DOI:10.1109/IPDRM51949.2020.00006
    8. Baorong Ding, Mingcong Han, Rong Chen. DArray: A High Performance RDMA-Based Distributed Array. Proceedings of the 52nd International Conference on Parallel Processing, DOI:10.1145/3605573.3605608

    Other cited types(0)

Catalog

    Get Citation
    PDF
    XML
    Article views (71) PDF downloads (770) Cited by(8)
    Related

    Copyright ©2025 JCST, All Rights Reserved     京ICP备05002829号-1

    Institute of Computing Technology, Chinese Academy of Sciences
    P.O. Box 2704, Beijing 100190, P.R. China

    Tel.: 86-10-62610746 E-mail: jcst@ict.ac.cn

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return