Abstract:Powered by advances in deep learning (DL) techniques, machine learning and artificial intelligence have achieved astonishing successes. However, the rapidly growing needs for DL also led to communication- and resource-intensive distributed training jobs for large-scale DL training, which are typically deployed over GPU clusters. To sustain the ever-increasing demand for DL training, the so-called "ring-all-reduce" (RAR) technologies have recently emerged as a favorable computing architecture to efficiently process network communication and computation load in GPU clusters. The most salient feature of RAR is that it removes the need for dedicated parameter servers, thus alleviating the potential communication bottleneck. However, when multiple RAR-based DL training jobs are deployed over GPU clusters, communication bottlenecks could still occur due to contentions between DL training jobs. So far, there remains a lack of theoretical understanding on how to design contention-aware resource scheduling algorithms for RAR-based DL training jobs, which motivates us to fill this gap in this work. Our main contributions are three-fold: i) We develop a new analytical model that characterizes both communication overhead related to the worker distribution of the job and communication contention related to the co-location of different jobs; ii) Based on the proposed analytical model, we formulate the problem as a non-convex integer program to minimize the makespan of all RAR-based DL training jobs. To address the unique structure in this problem that is not amenable for optimization algorithm design, we reformulate the problem into an integer linear program that enables provable approximation algorithm design called SJF-BCO (Smallest Job First with Balanced Contention and Overhead); and iii) We conduct extensive experiments to show the superiority of SJF-BCO over existing schedulers.

Mirage: Towards Low-interruption Services on Batch GPU Clusters with Reinforcement Learning

SCHED²: Scheduling Deep Learning Training Via Deep Reinforcement Learning.

A Dual-Agent Scheduler for Distributed Deep Learning Jobs on Public Cloud Via Reinforcement Learning

Liquid: Intelligent Resource Estimation and Network-Efficient Scheduling for Deep Learning Jobs on Distributed GPU Clusters

ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG

Multi-resource Interleaving for Deep Learning Training

Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing

Horus: An Interference-Aware Resource Manager for Deep Learning Systems

On Scheduling Ring-All-Reduce Learning Jobs in Multi-Tenant GPU Clusters with Communication Contention

Multi-Stream Scheduling of Inference Pipelines on Edge Devices - a DRL Approach

Multi-Agent Deep Reinforcement Learning-Based Resource Allocation in HPC/AI Converged Cluster

Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU

On a Meta Learning-based Scheduler for Deep Learning Clusters

DistMind: Efficient Resource Disaggregation for Deep Learning Workloads

Hierarchical Resource Partitioning on Modern GPUs: A Reinforcement Learning Approach

DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters

RLScheduler: An Automated HPC Batch Job Scheduler Using Reinforcement Learning

Optimal Scheduling in IoT-Driven Smart Isolated Microgrids Based on Deep Reinforcement Learning

GMI-DRL: Empowering Multi-GPU Deep Reinforcement Learning with GPU Spatial Multiplexing

RLPTO : A reinforcement learning-based performance-time optimized task and resource scheduling mechanism for distributed machine learning

Plume: A Framework for High Performance Deep RL Network Controllers via Prioritized Trace Sampling