Abstract:The growth of Large Language Models (LLMs) has necessitated large-scale distributed training. Highly optimized frameworks, however, still suffer significant losses in Model FLOPS utilization (often below 50%) due to large communication volumes. Meanwhile, our comprehensive profiling shows that the computation- and communication-intensive operators overlap well. This paper introduces DHelix, a novel micro-structure that dramatically improves the efficiency of LLM training inspired by the DNA structure. Central to DHelix's design is Strand Interleaving (SI), which views the continuous stream of training micro-batches through a GPU as two strands. DHelix juxtaposes the forward and backward passes of the two strands and performs a systematic optimization for an SI plan that co-schedules the operators from the opposite strands, enabled by operator-level overlap profiling results and a dynamic-programming based search algorithm. Meanwhile, DHelix enables the two strands to share model states and space for activation data, effectively accommodating two micro-batches with under 3% extra memory space. Dhelix seamlessly integrates with all forms of existing data/model parallelism, the most challenging being pipeline parallelism, thanks to its unique model folding design that results in a W-shaped pipeline. We evaluate DHelix training with the popular Llama and GPT dense models, plus the Phi Mixture of Expert (MoE) model, across 3 GPU clusters (A40, A800, and H100). Results show that it achieves 12-40% (up to 58% MFU) and 2-29% (up to 71% MFU) improvement on the 64-A40 and 64-A800 clusters, respectively, significantly outperforming state-of-the-art methods. On the H100 cluster, though the faster network reduces DHelix's profit margin, it makes cross-node tensor parallelism promising, a practice currently prohibitive due to communication costs.

Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning

US-Byte: an Efficient Communication Framework for Scheduling Unequal-Sized Tensor Blocks in Distributed Deep Learning

Towards Efficient Scheduling of Federated Mobile Devices under Computational and Statistical Heterogeneity

A Multidimensional Communication Scheduling Method for Hybrid Parallel DNN Training

Boosting Large-scale Parallel Training Efficiency with C4: A Communication-Driven Approach

CO2: Efficient Distributed Training with Full Communication-Computation Overlap

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

High-Speed Data Communication with Advanced Networks in Large Language Model Training

Federated Learning-Based Cooperative Model Training for Task-Oriented Semantic Communication

Efficient Parallelization Layouts for Large-Scale Distributed Model Training

On Optimizing the Communication of Model Parallelism

Automated Tensor Model Parallelism with Overlapped Communication for Efficient Foundation Model Training

Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression

Decentralized Training of Foundation Models in Heterogeneous Environments

Scalable Heterogeneous Scheduling Based Model Parallelism for Real-Time Inference of Large-Scale Deep Neural Networks

How Useful is Communication Scheduling for Distributed Training?

ACCO: Accumulate while you Communicate, Hiding Communications in Distributed LLM Training

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Accelerating the Training of Large Language Models Using Efficient Activation Rematerialization and Optimal Hybrid Parallelism.

Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution