Abstract:To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system designed and implemented for training DNNs using geo-distributed GPUs across different computing clusters or individual devices. Decentralized training faces significant challenges regarding system design and efficiency, including: 1) the need for remote automatic differentiation (RAD), 2) support for flexible model definitions and heterogeneous software, 3) heterogeneous hardware leading to low resource utilization or the straggler problem, and 4) slow network communication. To address these challenges, in the system design, we represent the model as a directed acyclic graph of operators (OP-DAG). Each node in the DAG represents the operator in the DNNs, while the edge represents the data dependency between operators. Based on this design, 1) users are allowed to customize any DNN without caring low-level operator implementation; 2) we enable the task scheduling with the more fine-grained sub-tasks, offering more optimization space; 3) a DAG runtime executor can implement RAD withour requiring the consistent low-level ML framework versions. To enhance system efficiency, we implement a workload estimator and design an OP-Fence scheduler to cluster devices with similar bandwidths together and partition the DAG to increase throughput. Additionally, we propose an AdaTopK compressor to adaptively compress intermediate activations and gradients at the slowest communication links. To evaluate the convergence and efficiency of our system and algorithms, we train ResNet-101 and GPT-2 on three real-world testbeds using 48 GPUs connected with 8 Mbps~10 Gbps networks. Experimental results demonstrate that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.

OF-WFBP: A near-optimal communication mechanism for tensor fusion in distributed deep learning

US-Byte: an Efficient Communication Framework for Scheduling Unequal-Sized Tensor Blocks in Distributed Deep Learning

Optimize Scheduling of Federated Learning on Battery-powered Mobile Devices

WBSP: Addressing Stragglers in Distributed Machine Learning with Worker-Busy Synchronous Parallel

MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning

Fast Convergence for Federated Learning in OFDMA Systems

OASR-WFBP: An overlapping aware start-up sharing gradient merging strategy for efficient communication in distributed deep learning

FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion

On Optimizing the Communication of Model Parallelism

FSP: Towards Flexible Synchronous Parallel Frameworks for Distributed Machine Learning

cuFSDAF: An Enhanced Flexible Spatiotemporal Data Fusion Algorithm Parallelized Using Graphics Processing Units

Optimizing DNN Compilation for Distributed Training with Joint OP and Tensor Fusion

Optimized Data Rate Allocation for Dynamic Sensor Fusion over Resource Constrained Communication Networks

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

Towards Faster And Better Federated Learning: A Feature Fusion Approach

FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloads

Optimal Distributed Finite-Time Fusion Method for Multi-Sensor Networks under Dynamic Communication Weight

ParallelFusion

DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression