vTensor-Based GPU Memory Management for Edge Deep Learning Training

Supporting real-time and privacy-preserving learning at the edge is emerging as a critical trend, bringing forth substantial challenges for deep learning (DL) training in the context of limited GPU (graphic processing unit) memory. Recent work has sought to address the limitations by swapping tensors between GPU memory and CPU memory. Unfortunately, their tensor-based memory management encounters additional overhead since the swapped tensors do not align with the actual memory demands, resulting in decreased throughput. This paper introduces a vTensor-based memory management approach designed to mitigate memory swapping overhead. Virtualized tensors, dubbed vTensors, are used to finely align memory swapping amounts with real-time memory demands. Firstly, we introduce an abstraction layer that virtualizes coarse-grained tensors to multiple finer-grained vTensors. Secondly, we propose the Layered Graph Model (LGM) for analyzing vTensor mappings, which produces a memory swapping plan leveraged in the subsequent DL training iterations. Evaluations conducted on typical edge deep learning models illustrate that our approach surpasses prior work with a 15.60% increase in DL training throughput.