Abstract:Transformer models have emerged as potent solutions to a wide array of multidisciplinary challenges. The deployment of Transformer architectures is significantly hindered by their extensive computational and memory requirements, necessitating the reliance on advanced efficient distributed training methodologies. Prior research has delved into the performance bottlenecks associated with distributed training, aiming to unravel these bottlenecks and suggest optimization directions. However, such analyses often overlook three aspects unique to Transformer models: the specialized architecture, the dependency on various distributed strategies, and the requirement to balance computational and memory overhead. This paper aims to bridge this gap by offering a comprehensive examination of the performance bottlenecks inherent in distributed training of Transformer models, leveraging both theoretical analysis and empirical investigation. We propose an analytical framework tailored to these unique aspects of Transformers, facilitating a holistic evaluation of model architectures, distributed strategies, and resource consumption. Based on this analytical framework, we conduct a comparative analysis of theoretical performances and further systematically explore how various distributed training strategies fare in real-world scenarios. Most of the experimental results can be well explained by the analytical outcomes derived from the analytical framework. Notably, our findings suggest an advantage of pipeline parallelism over data parallelism for Transformer models. Moreover, we shed light on some unexpected outcomes, such as the potential for increased total memory overhead due to suboptimal model partitioning within pipeline parallelism. Additionally, we underscore the significance of communication block size and waiting time to further enhance performance.

Galvatron

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

Improving Automatic Parallel Training via Balanced Memory Workload Optimization

An Efficient 2D Method for Training Super-Large Deep Learning Models

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

H3T: Efficient Integration of Memory Optimization and Parallelism for High-Throughput Transformer Training

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

On the Performance and Memory Footprint of Distributed Training: An Empirical Study on Transformers

H3T: Efficient Integration of Memory Optimization and Parallelism for Large-scale Transformer Training.

PrimePar: Efficient Spatial-temporal Tensor Partitioning for Large Transformer Model Training

3D Parallelism for Transformers Via Integer Programming

Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping

MDP: Model Decomposition and Parallelization of Vision Transformer for Distributed Edge Inference

TorchGT: A Holistic System for Large-scale Graph Transformer Training

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers

Efficient Large-Scale Language Model Training on GPU Clusters

TorchScale: Transformers at Scale

Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors

What comes after transformers? -- A selective survey connecting ideas in deep learning