Auto-SpMM: Towards Efficient Sparse Matrix-Matrix Multiplication on Heterogeneous Servers via Automatic Resource Mapping and Pipeline Generation

Sparse matrix-matrix multiplication (SpMM) is the core computing routine in many advanced domains, including artificial intelligence and scientific computing. Thus, accelerating SpMM has become a focal area for high-performance computing (HPC) researchers. While many advanced studies attempt to optimize the SpMM computation kernels on GPUs, the overall SpMM performance on GPUs cannot achieve consistent speedups over CPU implementations, due to time-consuming data transfer between the host and the device. Moreover, although SpMM on GPUs can resort to a pipeline for computation-communication overlap, whether an arbitrary sparse matrix can benefit from such a pipeline remains undetermined. To address these challenges, we propose Auto-SpMM, which is an efficient auto-tuning scheme for SpMM on a CPU-GPU heterogeneous server. Auto-SpMM extracts concise features that capture the memory-access and computation pattern of sparse matrices, and then automatically selects the optimal computation resource and algorithm, along with the best pipeline pattern for each SpMM. Extensive evaluations with 2 644 sparse matrices demonstrate that Auto-SpMM can achieve an average speedup of 9.71x compared with Intel Math Kernel Library (MKL) on CPUs and 13.20x compared with cuSPARSE on GPUs.