Abstract:With the rapid growth in the volume of data sets, models, and devices in the domain of deep learning, there is increasing attention on large-scale distributed deep learning. In contrast to traditional distributed deep learning, the large-scale scenario poses new challenges that include fault tolerance, scalability of algorithms and infrastructures, and heterogeneity in data sets, models, and resources. Due to intensive synchronization of models and sharing of data across GPUs and computing nodes during distributed training and inference processes, communication efficiency becomes the bottleneck for achieving high performance at a large scale. This article surveys the literature over the period of 2018-2023 on algorithms and technologies aimed at achieving efficient communication in large-scale distributed deep learning at various levels, including algorithms, frameworks, and infrastructures. Specifically, we first introduce efficient algorithms for model synchronization and communication data compression in the context of large-scale distributed training. Next, we introduce efficient strategies related to resource allocation and task scheduling for use in distributed training and inference. After that, we present the latest technologies pertaining to modern communication infrastructures used in distributed deep learning with a focus on examining the impact of the communication overhead in a large-scale and heterogeneous setting. Finally, we conduct a case study on the distributed training of large language models at a large scale to illustrate how to apply these technologies in real cases. This article aims to offer researchers a comprehensive understanding of the current landscape of large-scale distributed deep learning and to reveal promising future research directions toward communication-efficient solutions in this scope.

Communication Patterns in Distributed Deep Learning

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

Performance Analysis and Comparison of Distributed Machine Learning Systems

Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey

A Quantitative Survey of Communication Optimizations in Distributed Deep Learning

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

Coded Parallelism for Distributed Deep Learning.

Communication optimization strategies for distributed deep neural network training: A survey

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

Demystifying the Communication Characteristics for Distributed Transformer Models

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

A Survey From Distributed Machine Learning to Distributed Deep Learning

Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters

A Survey of Distributed Learning in Cloud, Mobile, and Edge Settings

Performance Modeling of Distributed Deep Neural Networks

A Linear Algebraic Approach to Model Parallelism in Deep Learning

Peering Beyond the Gradient Veil with Distributed Auto Differentiation

Communication Compression Techniques in Distributed Deep Learning: A Survey

Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging