Abstract:To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system designed and implemented for training DNNs using geo-distributed GPUs across different computing clusters or individual devices. Decentralized training faces significant challenges regarding system design and efficiency, including: 1) the need for remote automatic differentiation (RAD), 2) support for flexible model definitions and heterogeneous software, 3) heterogeneous hardware leading to low resource utilization or the straggler problem, and 4) slow network communication. To address these challenges, in the system design, we represent the model as a directed acyclic graph of operators (OP-DAG). Each node in the DAG represents the operator in the DNNs, while the edge represents the data dependency between operators. Based on this design, 1) users are allowed to customize any DNN without caring low-level operator implementation; 2) we enable the task scheduling with the more fine-grained sub-tasks, offering more optimization space; 3) a DAG runtime executor can implement RAD withour requiring the consistent low-level ML framework versions. To enhance system efficiency, we implement a workload estimator and design an OP-Fence scheduler to cluster devices with similar bandwidths together and partition the DAG to increase throughput. Additionally, we propose an AdaTopK compressor to adaptively compress intermediate activations and gradients at the slowest communication links. To evaluate the convergence and efficiency of our system and algorithms, we train ResNet-101 and GPT-2 on three real-world testbeds using 48 GPUs connected with 8 Mbps~10 Gbps networks. Experimental results demonstrate that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.

swFLOW: A large-scale distributed framework for deep learning on Sunway TaihuLight supercomputer

Distributed deep learning system for cancerous region detection on Sunway TaihuLight

swCaffe: a Parallel Framework for Accelerating Deep Learning Applications on Sunway TaihuLight

Swdnn: A Library for Accelerating Deep Learning Applications on Sunway TaihuLight.

Analysis and MPI Implementation of LQCD Dslash on Sunway TaihuLight*

Optimizing Convolutional Neural Networks on the Sunway TaihuLight Supercomputer.

swHPFM: Refactoring and Optimizing the Structured Grid Fluid Mechanical Algorithm on the Sunway TaihuLight Supercomputer

SoCFlow: Efficient and Scalable DNN Training on SoC-Clustered Edge Servers

Bridging the Gap between Deep Learning and Frustrated Quantum Spin System for Extreme-scale Simulations on New Generation of Sunway Supercomputer

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

Automatic Deep Learning Operator Fusion on Sunway SW26010 Many-Core Processor

Scaling The Training Of Recurrent Neural Networks On Sunway Taihulight Supercomputer

Swtensor: Accelerating Tensor Decomposition on Sunway Architecture

FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters

swCUDA: Auto parallel code translation framework from CUDA to ATHREAD for new generation sunway supercomputer

swTVM: Towards Optimized Tensor Code Generation for Deep Learning on Sunway Many-Core Processor

Towards a Peta-scale Unstructured Computational Fluid Dynamics (CFD) Acceleration Toolkit Based on Sunway Architectures?

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression

swATOP: Automatically Optimizing Deep Learning Operators on SW26010 Many-Core Processor

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training