Abstract:Gradient compression is a promising approach to alleviating the communication bottleneck in data parallel deep neural network (DNN) training by significantly reducing the data volume of gradients for synchronization. While gradient compression is being actively adopted by the industry (e.g., Facebook and AWS), our study reveals that there are two critical but often overlooked challenges: 1) inefficient coordination between compression and communication during gradient synchronization incurs substantial overheads, and 2) developing, optimizing, and integrating gradient compression algorithms into DNN systems imposes heavy burdens on DNN practitioners, and ad-hoc compression implementations often yield surprisingly poor system performance. In this paper, we propose a compression-aware gradient synchronization architecture, CaSync, which relies on flexible composition of basic computing and communication primitives. It is general and compatible with any gradient compression algorithms and gradient synchronization strategies and enables high-performance computation-communication pipelining. We further introduce a gradient compression toolkit, CompLL, to enable efficient development and automated integration of on-GPU compression algorithms into DNN systems with little programming burden. Lastly, we build a compression-aware DNN training framework HiPress with CaSync and CompLL. HiPress is open-sourced and runs on mainstream DNN systems such as MXNet, TensorFlow, and PyTorch. Evaluation via a 16-node cluster with 128 NVIDIA V100 GPUs and a 100 Gbps network shows that HiPress improves the training speed over current compression-enabled systems (e.g., BytePS-onebit, Ring-DGC and PyTorch-PowerSGD) by 9.8%-69.5% across six popular DNN models.

Accelerating Tensor Swapping in GPUs with Self-Tuning Compression

CSWAP: A Self-Tuning Compression Framework for Accelerating Tensor Swapping in GPUs

A Swap Dominated Tensor Re-Generation Strategy for Training Deep Learning Models

Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks

SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget

STR: Hybrid Tensor Re-Generation to Break Memory Wall for DNN Training

Hybrid Tensor Decomposition in Neural Network Compression

Efficient Memory Management for GPU-based Deep Learning Systems

A Generic, High-Performance, Compression-Aware Framework for Data Parallel DNN Training

pommDNN: Performance optimal GPU memory management for deep neural network training

FreeLunch: Compression-based GPU Memory Management for Convolutional Neural Networks

ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition

G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations

Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping

Mixed-TD: Efficient Neural Network Accelerator with Layer-Specific Tensor Decomposition

ASC: Adaptive Scale Feature Map Compression for Deep Neural Network

Swtensor: Accelerating Tensor Decomposition on Sunway Architecture

CODAG: Characterizing and Optimizing Decompression Algorithms for GPUs

TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training Via Tensor Splitting

Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework

Efficient Compression of Sparse Accelerator Data Using Implicit Neural Representations and Importance Sampling