Containerlet: Software-Granular Containers for High-Performance Deployment of Machine Learning Applications

Large-scale machine learning (ML) clusters deploy hundreds of models packaged as large container images, making scheduling and image pulling dominant contributors to deployment latency. Current systems adopt a hybrid-granularity strategy: they schedule using image-level metadata, reuse on-node layer caches, and lazily fetch content at the file or chunk level. This fragmentation leads to suboptimal cross-image reuse. We prove that an intermediate unit, software granularity, can be applied uniformly at scheduling, caching, and pulling to improve deployment performance. We present Containerlets, software-scoped layers that reorganize standard images into one private layer plus multiple reusable per-software units that reside in compatible registries and are pulled by unmodified runtimes. Containerlets promote cross-image reuse while bounding the number of units the scheduler must consider. We design a file backtracking method to convert existing images into Containerlet images, and we extend Kubernetes and the container runtime to support software-granular scheduling, caching, and lazy pulling. Experiments across diverse ML images and cluster traces show consistent gains, reducing deployment overhead by 18% compared with state-of-the-art systems. By treating software as a first-class unit of composition and reuse, Containerlets bridge compatibility and performance and provide a practical path to faster, more predictable ML deployment without changes to the container ecosystem.