Self-Adaptive Resource Management for Large-Scale Shared Clusters

In a shared cluster, each application runs on a subset of nodes and these subsets can overlap with one another. Resource management in such a cluster should adaptively change the application placement and workload assignment to satisfy the dynamic applications workloads and optimize the resource usage. This becomes a challenging problem with the cluster scale and application amount growing large. This paper proposes a novel self-adaptive resource management approach which is inspired from human market: the nodes trade their shares of applications' requests with others via auction and bidding to decide its own resource allocation and a global high-quality resource allocation is achieved as an emergent collective behavior of the market. Experimental results show that the proposed approach can ensure quick responsiveness, high scalability, and application prioritization in addition to managing the resources effectively.