Abstract:Distributed deep learning frameworks facilitate large deep learning workloads. These frameworks support sharing one GPU device among multiple jobs to improve resource utilization. Modern deep learning training jobs consume a large amount of GPU memory. Despite that, sharing GPU memory among jobs is still possible because a training job has iterative steps that its memory usage fluctuates over time. However, resource sharing also introduces the risk of job performance degradation. Co-located jobs sharing a GPU device may suffer from different levels of interference, mainly caused by memory oversharing. How to improve resource utilization while maintaining good job performance is a novel challenge for job placement strategies. This paper studies the job placement problem. We propose an opportunistic memory sharing model to describe the time-varying job memory requirements. Based on this model, we introduce an Opportunistic Job Placement Problem (OJPP) for shared GPU clusters that seek job placement configurations using a minimum number of GPU devices and guarantee user-defined performance requirements at the same time. We propose a greedy algorithm and a heuristic algorithm with computational complexities of O(n log n) and O(n2log n), respectively, to solve the problem. We also propose an online adjustment algorithm with the computational complexity of O(n log n) to perform updates to job placement configurations in runtime. A machine-learning-based interference prediction method is used to prepare accurate interference estimations. Extensive experiments are conducted on a GPU cluster to verify the correctness and effectiveness of our algorithms. Compared with standalone training jobs on dedicated clusters, the proposed approach reduces resource consumption by 46% in a shared cluster, while guaranteeing over 92.97% of the job performance, in terms of average job completion time.

Sampling-Based Multi-Job Placement for Heterogeneous Deep Learning Clusters

On-Line Scheduling of Parallel Jobs in Heterogeneous Multiple Clusters

Scheduling Distributed Deep Learning Jobs in Heterogeneous Cluster with Placement Awareness

Online Scheduling Algorithm for Heterogeneous Distributed Machine Learning Jobs

SchedTune: A Heterogeneity-Aware GPU Scheduler for Deep Learning

ReLoca: Optimize Resource Allocation for Data-parallel Jobs Using Deep Learning

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Online Job Scheduling in Distributed Machine Learning Clusters

The Case for Task Sampling based Learning for Cluster Job Scheduling

Hadar: Heterogeneity-Aware Optimization-Based Online Scheduling for Deep Learning Cluster

Grouper: Accelerating Hyperparameter Searching in Deep Learning Clusters with Network Scheduling

Large-scale Machine Learning Cluster Scheduling via Multi-agent Graph Reinforcement Learning

Interference-Aware Opportunistic Job Placement for Shared Distributed Deep Learning Clusters

Task allocation for decentralized training in heterogeneous environment

A Near Optimal Multi-Faced Job Scheduler For Datacenter Workloads

JPAS: Job-progress-aware Flow Scheduling for Deep Learning Clusters

Online Scheduling of Distributed Machine Learning Jobs for Incentivizing Sharing in Multi-Tenant Systems

A Reinforcement Learning Based Job Scheduling Algorithm for Heterogeneous Computing Environment

Learning to Optimize DAG Scheduling in Heterogeneous Environment

On a Meta Learning-based Scheduler for Deep Learning Clusters

Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing