Abstract:Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) are the three strategies widely adopted to enable fast and efficient Large Language Model (LLM) training. However, these approaches rely on data-intensive communication routines to collect, aggregate, and re-distribute gradients, activations, and other important model information, which pose significant overhead. Co-designed with GPU-based compression libraries, MPI libraries have been proven to reduce message size significantly, and leverage interconnect bandwidth, thus increasing training efficiency while maintaining acceptable accuracy. In this work, we investigate the efficacy of compression-assisted MPI collectives under the context of distributed LLM training using 3D parallelism and ZeRO optimizations. We scaled up to 192 V100 GPUs on the Lassen supercomputer. First, we enabled a naïve compression scheme across all collectives and observed a 22.5\% increase in TFLOPS per GPU and a 23.6\% increase in samples per second for GPT-NeoX-20B training. Nonetheless, such a strategy ignores the sparsity discrepancy among messages communicated in each parallelism degree, thus introducing more errors and causing degradation in training loss. Therefore, we incorporated hybrid compression settings toward each parallel dimension and adjusted the compression intensity accordingly. Given their low-rank structure (<a class="link-https" data-arxiv-id="2301.02654" href="https://arxiv.org/abs/2301.02654">arXiv:2301.02654</a>), we apply aggressive compression on gradients when performing DP All-reduce. We adopt milder compression to preserve precision while communicating activations, optimizer states, and model parameters in TP and PP. Using the adjusted hybrid compression scheme, we demonstrate a 17.3\% increase in TFLOPS per GPU and a 12.7\% increase in samples per second while reaching baseline loss convergence.

Parallelizing Training of Deep Generative Models on Massive Scientific Datasets

Efficient Large-Scale Language Model Training on GPU Clusters

Data-parallel distributed training of very large models beyond GPU capacity

An Efficient 2D Method for Training Super-Large Deep Learning Models

Optimizing Distributed Training on Frontier for Large Language Models

A New Distributed Method for Training Generative Adversarial Networks

Decentralized Training of Foundation Models in Heterogeneous Environments

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression

A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs

Scaling the training of particle classification on simulated MicroBooNE events to multiple GPUs

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Data

Parallel Inference for Latent Dirichlet Allocation on Graphics Processing Units.

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

Scalable training of graph convolutional neural networks for fast and accurate predictions of HOMO-LUMO gap in molecules

Highly-scalable, physics-informed GANs for learning solutions of stochastic PDEs

Accelerating Large Language Model Training with Hybrid GPU-based Compression

Towards Training Billion Parameter Graph Neural Networks for Atomic Simulations

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

Distributed Training Large-Scale Deep Architectures

Training Deep Surrogate Models with Large Scale Online Learning