Containerlet: Software-Granular Container Layer for High-Performance Deployment of Machine Learning Applications

Large-scale machine learning (ML) clusters deploy hundreds of models packaged as large container images, making scheduling and image pulling dominant contributors to deployment latency. Current systems adopt a hybrid-granularity strategy: they schedule using image-level metadata, reuse on-node layer caches, and lazily fetch content at the file or chunk level. This fragmentation leads to suboptimal cross-image reuse. We show that an intermediate unit, software granularity, can be uniformly applied at scheduling, caching, and pulling to improve deployment performance. We present Containerlet, a software-scoped layer that reorganizes standard images into one private layer plus multiple reusable per-software units that reside in compatible registries and are pulled by unmodified runtimes. Containerlet promotes cross-image reuse while bounding the number of units that the scheduler must consider. We design a file backtracking method to convert standard images into Containerlet images, which includes one private layer and multiple Containerlets. We also extend Kubernetes and container runtime to support software-granular scheduling, caching, and lazy pulling. End-to-end deployment experiments on diverse ML images and production cluster traces show consistent gains, reducing deployment overhead by 18% compared with state-of-the-art systems.