vTensor-based GPU Memory Management for Edge Deep Learning Training

Supporting real-time and privacy-preserving learning at the edge is emerging as a critical trend, bringing forth substantial challenges for deep learning (DL) training in the context of limited GPU memory. Recent work has sought to address the limitations by swapping tensors between GPU memory and CPU memory. Unfortunately, their tensor-based memory management encounters additional overhead since the swapped tensors do not align with the actual memory demands, resulting in decreased throughput. This paper introduces a vTensor-based memory management designed to mitigate memory swapping overhead. Firstly, we introduce an abstraction layer that virtualizes coarse-grained tensors to multiple finer-grained vTensors. Secondly, we propose Layered Graph Model (LGM) for analyzing vTensor mappings, which produces a memory swapping plan leveraged in the subsequent DL training iterations. Evaluations conducted on typical edge deep learning models illustrate that our approach surpasses prior work with a 15.60% increase in DL training throughput.