Abstract:The transformer-based deep neural network (DNN) models have shown considerable success across diverse tasks, prompting widespread adoption of distributed training methods such as data parallelism and pipeline parallelism. With the increasing parameter number, hybrid parallel training becomes imperative to scale training. The primary bottleneck in scaling remains the communication overhead. The communication scheduling technique, emphasizing the overlap of communication with computation, has demonstrated its benefits in scaling. However, most existing works focus on data parallelism, overlooking the nuances of hybrid parallel training. In this paper, we propose TriRace , an efficient communication scheduling framework for accelerating communications in hybrid parallel training of asynchronous pipeline parallelism and data parallelism. To achieve effective computation-communication overlap, TriRace introduces 3D communication scheduling , which adeptly leverages data dependencies between communication and computations, efficiently scheduling AllReduce communication, sparse communication, and peer-to-peer communication in hybrid parallel training. To avoid possible communication contentions, TriRace also incorporates a topology-aware runtime which optimizes the execution of communication operations by considering ongoing communication operations and real-time network status. We have implemented a prototype of TriRace based on PyTorch and Pipedream-2BW, and conducted comprehensive evaluations with three representative baselines. Experimental results show that TriRace achieves up to 1.07–1.45× speedup compared to the state-of-the-art pipeline parallelism training baseline Pipedream-2BW, and 1.24–1.81× speedup compared to the Megatron.

A generic communication scheduler for distributed DNN training acceleration

US-Byte: an Efficient Communication Framework for Scheduling Unequal-Sized Tensor Blocks in Distributed Deep Learning

AutoByte: Automatic Configuration for Optimal Communication Scheduling in DNN Training

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Accelerating Distributed DNN Training via Transport Layer Scheduling

A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters

A Multidimensional Communication Scheduling Method for Hybrid Parallel DNN Training

Prophet: Speeding Up Distributed DNN Training with Predictable Communication Scheduling.

Mercury: A Simple Transport Layer Scheduler to Accelerate Distributed DNN Training

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

BCEdge: SLO-Aware DNN Inference Services With Adaptive Batch-Concurrent Scheduling on Edge Devices

Effective Scheduler for Distributed DNN Training Based on MapReduce and GPU Cluster

DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling

Exploiting Simultaneous Communications to Accelerate Data Parallel Distributed Deep Learning

dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training

BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platforms

Scalable Heterogeneous Scheduling Based Model Parallelism for Real-Time Inference of Large-Scale Deep Neural Networks

QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices

Optimizing Task Placement and Online Scheduling for Distributed GNN Training Acceleration

DynaHB: A Communication-Avoiding Asynchronous Distributed Framework with Hybrid Batches for Dynamic GNN Training

Aries: A DNN Inference Scheduling Framework for Multi-core Accelerators