Abstract:In recent years， containers have gradually replaced virtual machines and are widely used in deep learning cloud platforms due to their lightweight and high scalability. However， the deep learning cloud platform still has deficiencies in GPU resource management， which are mainly manifested as multiple containers cannot share GPU resources due to the limitation of container orchestration technology. For some small-scale model training tasks and model inference tasks， a single task cannot fully utilize the computing resources of the entire GPU card. The current exclusive mode will result in a waste of expensive GPU resources， reduce resource efficiency and service availability. In response to this problem， this paper proposes a GPU sharing scheduling system. On the one hand， the Kubernetes-based Operator mechanism extends the existing cluster functions， enabling multiple Pods to share GPU resources， and designs an agent mechanism to ensure that compatibility with native Kubernetes. On the other hand， based on the GPU time slice and preemption mechanism， the dynamic management and scheduling of GPU resources is realized， fine-grained coordination among multiple tasks is performed， and task interference is reduced. Experimental results show that compared with the native Kubernetes scheduling system， the proposed system can reduce the completion time of a group of deep learning training tasks by about 20% on average， and increase the utilization of cluster GPU resources by about 10% on average. When the GPU is shared， the performance of high-priority tasks is less than 5% compared to the exclusive GPU， and the low-priority tasks can run on the same GPU with 20% of the performance.

Kale: Elastic GPU Scheduling for Online DL Model Training

SchedTune: A Heterogeneity-Aware GPU Scheduler for Deep Learning

Elastic Scheduler: Heterogeneous and Dynamic Deep Learning in the Cloud.

ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG

Liquid: Intelligent Resource Estimation and Network-Efficient Scheduling for Deep Learning Jobs on Distributed GPU Clusters

Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters

GPU Shared Scheduling System Under Deep Learning Container Cloud Platform

EasyScale: Elastic Training with Consistent Accuracy and Improved Utilization on GPUs

Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision

GAI: A Centralized Tree-Based Scheduler for Machine Learning Workload in Large Shared Clusters.

Effective Elastic Scaling of Deep Learning Workloads

Deep Learning Workload Scheduling in GPU Datacenters: A Survey

SCHED²: Scheduling Deep Learning Training Via Deep Reinforcement Learning.

Poster Abstract: Deep Learning Workloads Scheduling with Reinforcement Learning on GPU Clusters

Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters

An Empirical Study on Low GPU Utilization of Deep Learning Jobs

EasyScale: Accuracy-consistent Elastic Training for Deep Learning

GPU Cluster Scheduling for Network-Sensitive Deep Learning

Deep Learning Research and Development Platform: Characterizing and Scheduling with QoS Guarantees on GPU Clusters.

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster

Dynamic Resource Allocation for Deep Learning Clusters with Separated Compute and Storage