Abstract:Efficient GPU scheduling is the key to minimizing the execution time of the Deep Learning (DL) training workloads. DL training system schedulers typically allocate a fixed number of GPUs to each job, which inhibits high resource utilization and often extends the overall training time. The recent introduction of schedulers that can dynamically reallocate GPUs has achieved better cluster efficiency. This dynamic nature, however, introduces additional overhead by terminating and restarting jobs or requires modification to the DL training frameworks.We propose and develop an efficient, non-intrusive GPU scheduling framework that employs a combination of an adaptive GPU scheduler and an elastic GPU allocation mechanism to reduce the completion time of DL training workloads and improve resource utilization. Specifically, the adaptive GPU scheduler includes a scheduling algorithm that uses training job progress information to determine the most efficient allocation and reallocation of GPUs for incoming and running jobs at any given time. The elastic GPU allocation mechanism works in concert with the scheduler. It offers a lightweight and nonintrusive method to reallocate GPUs based on a “SideCar” process that temporarily stops and restarts the job’s DL training process with a different number of GPUs. We implemented the scheduling framework as plugins in Kubernetes and conducted evaluations on two 16-GPU clusters with multiple training jobs based on TensorFlow. Results show that our proposed scheduling framework reduces the overall execution time and the average job completion time by up to 45% and 63%, respectively, compared to the Kubernetes default scheduler. Compared to a termination based scheduler, our framework reduces the overall execution time and the average job completion time by up to 20% and 37%, respectively.

Effective Elastic Scaling of Deep Learning Workloads

ElasticDL: A Kubernetes-native Deep Learning Framework with Fault-tolerance and Elastic Scheduling

A Predictive Autoscaler for Elastic Batch Jobs

An Efficient and Non-Intrusive GPU Scheduling Framework for Deep Learning Training Systems.

Elan: Towards Generic and Efficient Elastic Training for Deep Learning

Elastic Scheduler: Heterogeneous and Dynamic Deep Learning in the Cloud.

ElasticFlow: an Elastic Serverless Training Platform for Distributed Deep Learning.

Dynamic Resource Allocation for Deep Learning Clusters with Separated Compute and Storage

Scaling Deep Learning on GPU and Knights Landing clusters

ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG

An Optimal Resource Allocator of Elastic Training for Deep Learning Jobs on Cloud

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

Elastic Deep Learning in Multi-Tenant GPU Clusters

DeepScaling

DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters

EasyScale: Accuracy-consistent Elastic Training for Deep Learning

E-LAS: Design and Analysis of Completion-Time Agnostic Scheduling for Distributed Deep Learning Cluster

Elastic Task Offloading and Resource Allocation over Hybrid Cloud: A Reinforcement Learning Approach

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

EasyScale: Elastic Training with Consistent Accuracy and Improved Utilization on GPUs

GPU Cluster Scheduling for Network-Sensitive Deep Learning