Abstract:To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system designed and implemented for training DNNs using geo-distributed GPUs across different computing clusters or individual devices. Decentralized training faces significant challenges regarding system design and efficiency, including: 1) the need for remote automatic differentiation (RAD), 2) support for flexible model definitions and heterogeneous software, 3) heterogeneous hardware leading to low resource utilization or the straggler problem, and 4) slow network communication. To address these challenges, in the system design, we represent the model as a directed acyclic graph of operators (OP-DAG). Each node in the DAG represents the operator in the DNNs, while the edge represents the data dependency between operators. Based on this design, 1) users are allowed to customize any DNN without caring low-level operator implementation; 2) we enable the task scheduling with the more fine-grained sub-tasks, offering more optimization space; 3) a DAG runtime executor can implement RAD withour requiring the consistent low-level ML framework versions. To enhance system efficiency, we implement a workload estimator and design an OP-Fence scheduler to cluster devices with similar bandwidths together and partition the DAG to increase throughput. Additionally, we propose an AdaTopK compressor to adaptively compress intermediate activations and gradients at the slowest communication links. To evaluate the convergence and efficiency of our system and algorithms, we train ResNet-101 and GPT-2 on three real-world testbeds using 48 GPUs connected with 8 Mbps~10 Gbps networks. Experimental results demonstrate that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.

ATOM: Asynchronous Training of Massive Models for Deep Learning in a Decentralized Environment

Decentralized Training of Foundation Models in Heterogeneous Environments

An Efficient 2D Method for Training Super-Large Deep Learning Models

A Multi-Level Framework for Accelerating Training Transformer Models

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

H3T: Efficient Integration of Memory Optimization and Parallelism for High-Throughput Transformer Training

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

H3T: Efficient Integration of Memory Optimization and Parallelism for Large-scale Transformer Training.

Distributed Training of Large Language Models

Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer

Efficient Large-Scale Language Model Training on GPU Clusters

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression

Evolving Subnetwork Training for Large Language Models

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

Optimizing Distributed Training on Frontier for Large Language Models

Evaluation of pre-training large language models on leadership-class supercomputers

From promise to practice: realizing high-performance decentralized training

Improving Automatic Parallel Training Via Balanced Memory Workload Optimization