Abstract:Powered by advances in deep learning (DL) techniques, machine learning and artificial intelligence have achieved astonishing successes. However, the rapidly growing needs for DL also led to communication- and resource-intensive distributed training jobs for large-scale DL training, which are typically deployed over GPU clusters. To sustain the ever-increasing demand for DL training, the so-called "ring-all-reduce" (RAR) technologies have recently emerged as a favorable computing architecture to efficiently process network communication and computation load in GPU clusters. The most salient feature of RAR is that it removes the need for dedicated parameter servers, thus alleviating the potential communication bottleneck. However, when multiple RAR-based DL training jobs are deployed over GPU clusters, communication bottlenecks could still occur due to contentions between DL training jobs. So far, there remains a lack of theoretical understanding on how to design contention-aware resource scheduling algorithms for RAR-based DL training jobs, which motivates us to fill this gap in this work. Our main contributions are three-fold: i) We develop a new analytical model that characterizes both communication overhead related to the worker distribution of the job and communication contention related to the co-location of different jobs; ii) Based on the proposed analytical model, we formulate the problem as a non-convex integer program to minimize the makespan of all RAR-based DL training jobs. To address the unique structure in this problem that is not amenable for optimization algorithm design, we reformulate the problem into an integer linear program that enables provable approximation algorithm design called SJF-BCO (Smallest Job First with Balanced Contention and Overhead); and iii) We conduct extensive experiments to show the superiority of SJF-BCO over existing schedulers.

New YARN sharing GPU based on graphics memory granularity scheduling

GPUhd: Augmenting YARN with GPU Resource Management

gShare: A centralized GPU memory management framework to enable GPU memory sharing for containers

New Scheduling Algorithms for Improving Performance and Resource Utilization in Hadoop YARN Clusters

A Virtual Multi-Channel GPU Fair Scheduling Method for Virtual Machines.

Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing

Push-based Network-efficient Hadoop YARN Scheduling Mechanism for In-memory Computing

Agglomerative Memory and Thread Scheduling for High-Performance Ray-Tracing on GPUs

HiveD: Sharing a GPU Cluster for Deep Learning with Guarantees

HyTGraph: GPU-Accelerated Graph Processing with Hybrid Transfer Management

Exploring the Diversity of Multiple Job Deployments over GPUs for Efficient Resource Sharing

Data Partitioning Strategy of GPU Heterogeneous Clusters Based on Learning

GPU Sharing with Triples Mode

G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

GPU Scheduling for Short Tasks in Private Cloud

MGPU-TSM: A Multi-GPU System with Truly Shared Memory

A perceptual and predictive batch-processing memory scheduling strategy for a CPU-GPU heterogeneous system

On Scheduling Ring-All-Reduce Learning Jobs in Multi-Tenant GPU Clusters with Communication Contention

On Global Resource Allocation in Clusters for Data Analytics.

GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching