Abstract:Distributed training is the de facto standard to scale up the training of deep learning models with multiple GPUs. Its performance bottleneck lies in communications for gradient synchronization. Although high tensor sparsity is widely observed, the optimal communication scheme to fully leverage sparsity is still missing. This paper aims to bridge this gap. We first analyze the characteristics of sparse tensors in popular models to understand the fundamentals of sparsity. We then systematically explore the design space of communication schemes for sparse tensors and find the optimal ones. These findings give a new understanding and inspire us to develop a holistic gradient synchronization system called Zen for sparse tensors. We demonstrate that Zen can achieve up to 5.09x speedup in communication time and up to $2.48\times$ speedup in training throughput compared to the state-of-the-art methods.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the communication bottleneck problem of gradient synchronization in distributed training. Specifically, the author focuses on the problem of excessive communication overhead caused by the need to frequently synchronize gradient tensors when training deep - learning models in a multi - GPU environment. Although tensors in deep - learning models are usually highly sparse (that is, most elements are zero), existing communication schemes fail to fully utilize this sparsity to optimize communication efficiency. #### Core problems of the paper 1. **Communication bottleneck**: With the improvement of hardware computing power, the frequency of model training increases, which leads to an increased burden on the communication system. However, the upgrade of the network system has not kept up with the improvement of computing power, further intensifying the tension between computing and communication. 2. **Insufficiency of existing schemes**: Current communication schemes such as Ring - Allreduce and BytePS perform well in some cases, but they assume that gradient tensors are dense and ignore the sparsity in practice. This results in low communication efficiency. 3. **Under - utilized sparsity potential**: Many deep - learning models exhibit high tensor sparsity during gradient synchronization, which can significantly reduce the amount of communication. However, existing methods do not fully consider this sparsity, resulting in sub - optimal communication performance. #### Solutions To address the above problems, the paper proposes the following research questions: - **How to design an optimal communication scheme to fully utilize sparsity?** To this end, the author conducts research in the following aspects: 1. **Analysis of sparse tensor characteristics**: By analyzing the characteristics of sparse tensors in popular deep - learning models, the basic laws of gradient sparsity are revealed. 2. **Exploration of the design space of communication schemes**: Systematically explore the design space of communication schemes suitable for sparse tensors and find the optimal scheme. 3. **Development of the ZEN system**: Based on the above research, a gradient synchronization system named ZEN is developed, which can achieve near - optimal communication time. ZEN eliminates the additional overhead caused by data dependence and ensures the efficiency of communication and the optimality of results by introducing a data - independent hierarchical hashing algorithm and an efficient coding scheme. #### Main contributions - **Characteristic analysis**: Analyze the characteristics of sparse tensors in different deep - learning models and reveal the root causes of gradient sparsity. - **Design space exploration**: Systematically explore the design space of sparse communication schemes and find the optimal scheme. - **ZEN system**: Develop a gradient synchronization system named ZEN, which can achieve near - optimal communication time in distributed training. - **Performance evaluation**: Verified by experiments, ZEN shows a significant throughput improvement in various training tasks and has obvious advantages over existing methods in communication time and training throughput. In summary, this paper solves the communication bottleneck problem of gradient synchronization in distributed training by in - depth analysis of the characteristics of sparse tensors, systematic exploration of the design space of communication schemes, and development of an efficient gradient synchronization system.

Empowering Distributed Training with Sparsity-driven Data Synchronization

OF-WFBP: A near-optimal communication mechanism for tensor fusion in distributed deep learning

Sparse Communication for Training Deep Networks

SparDL: Distributed Deep Learning Training with Efficient Sparse Communication

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

SAP-SGD: Accelerating Distributed Parallel Training with High Communication Efficiency on Heterogeneous Clusters

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks

Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey

OSP: Boosting Distributed Model Training with 2-Stage Synchronization

Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning

Sparse Gradient Compression For Distributed Sgd

SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training

RedSync : Reducing Synchronization Traffic for Distributed Deep Learning

Global Momentum Compression for Sparse Communication in Distributed Learning

Robust Fully-Asynchronous Methods for Distributed Training over General Architecture

Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

Heter-Train: A Distributed Training Framework Based on Semi-Asynchronous Parallel Mechanism for Heterogeneous Intelligent Transportation Systems

A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration

Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent

RedSync: Reducing Synchronization Bandwidth for Distributed Deep Learning Training System