Abstract:Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) are the three strategies widely adopted to enable fast and efficient Large Language Model (LLM) training. However, these approaches rely on data-intensive communication routines to collect, aggregate, and re-distribute gradients, activations, and other important model information, which pose significant overhead. Co-designed with GPU-based compression libraries, MPI libraries have been proven to reduce message size significantly, and leverage interconnect bandwidth, thus increasing training efficiency while maintaining acceptable accuracy. In this work, we investigate the efficacy of compression-assisted MPI collectives under the context of distributed LLM training using 3D parallelism and ZeRO optimizations. We scaled up to 192 V100 GPUs on the Lassen supercomputer. First, we enabled a naïve compression scheme across all collectives and observed a 22.5\% increase in TFLOPS per GPU and a 23.6\% increase in samples per second for GPT-NeoX-20B training. Nonetheless, such a strategy ignores the sparsity discrepancy among messages communicated in each parallelism degree, thus introducing more errors and causing degradation in training loss. Therefore, we incorporated hybrid compression settings toward each parallel dimension and adjusted the compression intensity accordingly. Given their low-rank structure (<a class="link-https" data-arxiv-id="2301.02654" href="https://arxiv.org/abs/2301.02654">arXiv:2301.02654</a>), we apply aggressive compression on gradients when performing DP All-reduce. We adopt milder compression to preserve precision while communicating activations, optimizer states, and model parameters in TP and PP. Using the adjusted hybrid compression scheme, we demonstrate a 17.3\% increase in TFLOPS per GPU and a 12.7\% increase in samples per second while reaching baseline loss convergence.

ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology

Accelerating Large Language Model Training with Hybrid GPU-based Compression

Zero Bubble Pipeline Parallelism

RTP: Rethinking Tensor Parallelism with Memory Deduplication

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Parallel Volume Rendering Based on LOD Method

Optimal distributed parallel algorithms for deep learning framework Tensorflow

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

Accelerating Heterogeneous Tensor Parallelism via Flexible Workload Control

High-Performance Tensor-Train Primitives Using GPU Tensor Cores

A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs

Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster

TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices

OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

PrimePar: Efficient Spatial-temporal Tensor Partitioning for Large Transformer Model Training

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

On Optimizing the Communication of Model Parallelism

Scalable CP Decomposition for Tensor Learning using GPU Tensor Cores

Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure