Journal of Computer Science and Technology ›› 2020, Vol. 35 ›› Issue (2): 433-452.doi: 10.1007/s11390-020-9693-0

• Special Section of ChinaSys 2019 • Previous Articles     Next Articles

Huge Page Friendly Virtualized Memory Management

Sai Sha1,2,3, Member, ACM, Jing-Yuan Hu1, Ying-Wei Luo1,2,3,*, Member, CCF, ACM, Xiao-Lin Wang1,2,3, Member, CCF, ACM, Zhenlin Wang4, Member, ACM   

  1. 1 School of Electronics Engineering and Computer Science, Peking University, Beijing 100871, China;
    2 Peng Cheng Laboratory, Shenzhen 518052, China;
    3 Shenzhen Key Laboratory for Cloud Computing Technology & Applications, School of Electronic and Computer Engineering, Peking University Shenzhen, Shenzhen 518000, China;
    4 Department of Computer Science, Michigan Technological University, Michigan 49246, U.S.A.
  • Received:2019-05-07 Revised:2019-10-14 Online:2020-03-05 Published:2020-03-18
  • Contact: Ying-Wei Luo
  • About author:Sai Sha is a Ph.D. candidate in the School of Electronics Engineering and Computer Science, Peking University, Beijing. Before that, he received his Bachelor's degree in computer science from Beijing Institute of Technology, Beijing, in 2018. His research interests include system software, virtualization, domestic operating system, and deep learning.
  • Supported by:
    The work was supported by the National Key Research and Development Program of China under Grant No. 2018YFB1003604, the National Natural Science Foundation of China under Grant Nos. 61472008, 61672053 and U1611461, Shenzhen Key Research Project under Grant No. JCYJ20170412150946024, the National Science Foundation of USA under Grant No. CSR-1618384, and Beijing Technological Program under Grant No. Z181100008918015.

With the rapid increase of memory consumption by applications running on cloud data centers, we need more efficient memory management in a virtualized environment. Exploiting huge pages becomes more critical for a virtual machine's performance when it runs large working set size programs. Programs with large working set sizes are more sensitive to memory allocation, which requires us to quickly adjust the virtual machine's memory to accommodate memory phase changes. It would be much more efficient if we could adjust virtual machines' memory at the granularity of huge pages. However, existing virtual machine memory reallocation techniques, such as ballooning, do not support huge pages. In addition, in order to drive effective memory reallocation, we need to predict the actual memory demand of a virtual machine. We find that traditional memory demand estimation methods designed for regular pages cannot be simply ported to a system adopting huge pages. How to adjust the memory of virtual machines timely and effectively according to the periodic change of memory demand is another challenge we face. This paper proposes a dynamic huge page based memory balancing system (HPMBS) for efficient memory management in a virtualized environment. We first rebuild the ballooning mechanism in order to dispatch memory in the granularity of huge pages. We then design and implement a huge page working set size estimation mechanism which can accurately estimate a virtual machine's memory demand in huge pages environments. Combining these two mechanisms, we finally use an algorithm based on dynamic programming to achieve dynamic memory balancing. Experiments show that our system saves memory and improves overall system performance with low overhead.

Key words: virtualization, huge page, ballooning, memory balancing

[1] Khalidi Y A, Talluri M, Nelson M N, Williams D. Virtual memory support for multiple pages. Technical Report, Sun Microsystems Laboratories, Inc., 1993.;jsessionid=AA2CA3D6205E02FDA1FC545D691C5C20?doi=,Sept. 2019.
[2] Arcangeli A. Transparent hugepage support. In Proc. the 2010 KVM Forum, August 2010.
[3] Wang X, Luo T, Hu J, Wang Z, Luo Y. Evaluating the impacts of hugepage on virtual machines. Science China Information Sciences, 2017, 60(1):Article No. 12103.
[4] Denning P J. The working set model for program behavior. In Proc. the 1st ACM Symposium on Operating System Principles, October 1967, Article No. 15.
[5] Hu J, Bai X, Sha S et al. Working set size estimation with hugepages in virtualization. In Proc. the 2018 IEEE ISPA/IUCC/BDCloud/SocialCom/SustainCom, Dec. 2018, pp.501-508.
[6] Hu J, Bai X, Sha S et al. HUB:Hugepage ballooning in kernel-based virtual machines. In Proc. International Symposium on Memory Systems, Oct. 2018, pp.31-37.
[7] Mattson R L, Gecsei J, Slutz D R, Traiger I L. Evaluation techniques for storage hierarchies. IBM Systems Journal, 1970, 9(2):78-117.
[8] Waldspurger C A, Park N, Garthwaite A T, Ahmad I. Efficient MRC construction with SHARDS. In Proc. the 13th USENIX Conference on File and Storage Technologies, February 2015, pp.95-110.
[9] Wang Z, Wang X, Hou F, Luo Y, Wang Z. Dynamic memory balancing for virtualization. ACM Transactions on Architecture and Code Optimization, 2016, 13(1):Article No. 2.
[10] Zhao W, Wang Z, Luo Y. Dynamic memory balancing for virtual machines. ACM SIGOPS Operating Systems Review, 2009, 43(3):37-47.
[11] Waldspurger C A. Memory resource management in VMware ESX server. ACM SIGOPS Operating Systems Review, 2002, 36(5):181-194.
[12] Zhao W, Jin X, Wang Z, Wang X, Luo Y, Li X. Low cost working set size tracking. In Proc. the 2011 USENIX Annual Technical Conference, June 2011, Article No. 14.
[13] Zhou P, Pandey V, Sundaresan J, Raghuraman A, Zhou Y, Kumar S. Dynamic tracking of page miss ratio curve for memory management. ACM SIGOPS Operating Systems Review, 2004, 38(5):177-188.
[14] Wires J, Ingram S, Drudi Z, Harvey N J, Warfield A, Data C. Characterizing storage workloads with counter stacks. In Proc. the 11th USENIX Symposium on Operating Systems Design and Implementation, October 2014, pp.335-349.
[15] Niu Q, Dinan J, Lu Q, Sadayappan P. PARDA:A fast parallel reuse distance analysis algorithm. In Proc. the 26th International Parallel and Distributed Processing Symposium, May 2012, pp.1284-1294.
[16] Tam D K, Azimi R, Soares L B, Stumm M. RapidMRC:Approximating L2 miss rate curves on commodity systems for online optimizations. In Proc. the 14th International Conference on Architectural Support for Programming Languages and Operating Systems, March 2009, pp.121-132.
[17] Xiang X, Bao B, Ding C, Gao Y. Linear-time modeling of program working set in shared cache. In Proc. the 2011 International Conference on Parallel Architectures and Compilation Techniques, October 2011, pp.350-360.
[18] Hu X, Wang X, Zhou L, Luo Y, Ding C, Wang Z. Kinetic modeling of data eviction in cache. In Proc. the 2016 USENIX Annual Technical Conference, June 2016, pp.351-364.
[19] Hu X, Wang X, Zhou L, Luo Y, Wang Z, Ding C, Ye C. Fast miss ratio curve modeling for storage cache. ACM Transactions on Storage, 2018, 14(2):Article No. 12.
[20] Xiao Z, Song W, Chen Q. Dynamic resource allocation using virtual machines for cloud computing environment. IEEE Transactions on Parallel and Distributed Systems, 2012, 24(6):1107-1117.
[21] Tasoulas E, Haugerund H, Begnum K. Bayllocator:A proactive system to predict server utilization and dynamically allocate memory resources using Bayesian networks and ballooning. In Proc. the 26th Large Installation System Administration Conference on Strategies, Tools, and Techniques, December 2012, pp.111-122.
[22] Gordon A, Hines M, Silva D, Ben-Yehuda M, Silva M, Lizarraga G. Ginkgo:Automated, application-driven memory overcommitment for cloud computing. In Proc. the 2011 Workshop on Runtime Environments/Systems, Layering, and Virtualized Environments, May 2011.
[23] Nitu V, Kocharyan A, Yaya H, Tchana A, Hagimont D, Astsatryan H. Working set size estimation techniques in virtualized environments:One size does not fit all. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2018, 2(1):Article No. 19.
[24] Liu H, Jin H, Liao X, Deng W, He B, Xu C. Hotplug or ballooning:A comparative study on dynamic memory management techniques for virtual machines. IEEE Transactions on Parallel and Distributed Systems, 2015, 26(5):1350-1363.
[1] Xue-Kai Du, Zhi-Hui Lu, Qiang Duan, Jie Wu, Cheng-Rong Wu. LTSS:Load-Adaptive Traffic Steering and Forwarding for Security Services in Multi-Tenant Cloud Datacenters [J]. , 2017, 32(6): 1265-1278.
[2] Zuo-Ning Chen, Kang Chen, Jin-Lei Jiang, Lu-Fei Zhang, Song Wu, Zheng-Wei Qi, Chun-Ming Hu, Yong-Wei Wu, Yu-Zhong Sun, Hong Tang, Ao-Bing Sun, Zi-Lu Kang. Evolution of Cloud Operating System: From Technology to Ecosystem [J]. , 2017, 32(2): 224-241.
[3] Fan-Fu Zhou, Ru-Hui Ma, Jian Li, Li-Xia Chen, Wei-Dong Qiu, Hai-Bing Guan. Optimizations for High Performance Network Virtualization [J]. , 2016, 31(1): 107-116.
[4] Shu-Sheng Guo, Zi-Mu Yuan, Ao-Bing Sun, Qiang Yue. A New ETL Approach Based on Data Virtualization [J]. , 2015, 30(2): 311-323.
[5] Zheng Cao, Xiao-Li Liu, ACM Qiang Li, Xiao-Bing Liu, ACM Zhan Wang, Xue-Jun An. An Intra-Server Interconnect Fabric for Heterogeneous Computing [J]. , 2014, 29(6): 976-988.
[6] Hai-Bo Chen (陈海波), Member, CCF, ACM, IEEE, Feng-Zhe Zhang (张逢喆), Member, CCF, ACM, Rong Chen (陈榕), Bin-Yu Zang (臧斌宇), Senior Member, CCF, Member, ACM and Pen-Chung Yew (游本中), Fellow, IEEE. Mercury: Combining Performance with Dependability Using Self-Virtualization [J]. , 2012, 27(1): 92-104.
[7] Dan Feng and Hai Jin. Massive Storage Systems [J]. , 2006, 21(5): 648-664 .
[8] Cheng-Chun Shu, Hai-Yan Yu, and Hao-Zhi Liu. BEAP: An End-User Agile Programming Paradigm for Business Applications [J]. , 2006, 21(4): 609-619 .
Full text



[1] Chen Shihua;. On the Structure of Finite Automata of Which M Is an(Weak)Inverse with Delay τ[J]. , 1986, 1(2): 54 -59 .
[2] Liu Mingye; Hong Enyu;. Some Covering Problems and Their Solutions in Automatic Logic Synthesis Systems[J]. , 1986, 1(2): 83 -92 .
[3] Wang Xuan; Lü Zhimin; Tang Yuhai; Xiang Yang;. A High Resolution Chinese Character Generator[J]. , 1986, 1(2): 1 -14 .
[4] Gao Qingshi; Zhang Xiang; Yang Shufan; Chen Shuqing;. Vector Computer 757[J]. , 1986, 1(3): 1 -14 .
[5] Jin Lan; Yang Yuanyuan;. A Modified Version of Chordal Ring[J]. , 1986, 1(3): 15 -32 .
[6] Zhang Cui; Zhao Qinping; Xu Jiafu;. Kernel Language KLND[J]. , 1986, 1(3): 65 -79 .
[7] Qu Yanwen;. AGDL: A Definition Language for Attribute Grammars[J]. , 1986, 1(3): 80 -91 .
[8] Wang Jianchao; Wei Daozheng;. An Effective Test Generation Algorithm for Combinational Circuits[J]. , 1986, 1(4): 1 -16 .
[9] Chen Zhaoxiong; Gao Qingshi;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[10] Zheng Guoliang; Li Hui;. The Design and Implementation of the Syntax-Directed Editor Generator(SEG)[J]. , 1986, 1(4): 39 -48 .

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
  Copyright ©2015 JCST, All Rights Reserved