Speculative Container Scheduling for Deep Learning Applications in a Kubernetes Cluster
Ying Mao,Yuqi Fu,Wenjia Zheng,Long Cheng,Qingzhi Liu,Dingwen Tao
DOI: https://doi.org/10.1109/jsyst.2021.3129974
IF: 4.802
2021-01-01
IEEE Systems Journal
Abstract:In the past decade, we have witnessed a dramatically increasing volume of data collected from various sources. To maximize utilization, various machine and deep learning models have been developed to study data. While data-driven applications improve countless products, hyperparameter tuning for the models is still a time-consuming and resource-intensive process. Cloud computing provides infrastructure support for the training of deep learning applications. The cloud service providers create an isolated virtual environment for clients who share physical resources, e.g., CPU and memory. On the cloud, resource management schemes are implemented to enable better sharing among users and boost system-wide performance. However, general scheduling approaches, such as spread priority and balanced resource schedulers, do not work well with deep learning workloads. In this article, we propose SpeCon, a novel container scheduler optimized for short-lived deep learning applications. Based on virtualized containers, such as Kubernetes and Docker, SpeCon analyzes the typical characteristics of training processes. We design a suite of algorithms to monitor the trainings progress and speculatively migrate the slow-growing models to release resources for fast-growing ones. Specifically, the extensive experiments demonstrate that SpeCon improves an individual jobs completion time by up to 41.5, 14.8 system-wide, and 24.7 in terms of makespan.
computer science, information systems,telecommunications,engineering, electrical & electronic,operations research & management science