Abstract:Gradient compression (e.g., gradient quantization and gradient sparsification) is a core technique in reducing communication costs in distributed learning systems. The recent trend of gradient compression is to use a varying number of bits across iterations, however, relying on empirical observations or engineering heuristics without a systematic treatment and analysis. To the best of our knowledge, a general dynamic gradient compression that leverages both quantization and sparsification techniques is still far from understanding. This paper proposes a novel Adaptively-Compressed Stochastic Gradient Descent (AC-SGD) strategy to adjust the number of quantization bits and the sparsification size with respect to the norm of gradients, the communication budget, and the remaining number of iterations. In particular, we derive an upper bound, tight in some cases, of the convergence error for arbitrary dynamic compression strategy. Then we consider communication budget constraints and propose an optimization formulation - denoted as the Adaptive Compression Problem (ACP) - for minimizing the deep model’s convergence error under such constraints. By solving the ACP, we obtain an enhanced compression algorithm that significantly improves model accuracy under given communication budget constraints. Finally, through extensive experiments on computer vision and natural language processing tasks on MNIST, CIFAR-10, CIFAR-100 and AG-News datasets, respectively, we demonstrate that our compression scheme significantly outperforms the state-of-the-art gradient compression methods in terms of mitigating communication costs.

DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning

DQSGD: DYNAMIC QUANTIZED STOCHASTIC GRADIENT DESCENT FOR COMMUNICATION-EFFICIENT DISTRIBUTED LEARNING

Quantized Epoch-SGD for Communication-Efficient Distributed Learning.

Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees

Killing Two Birds with One Stone: Quantization Achieves Privacy in Distributed Learning

Truncated Non-Uniform Quantization for Distributed SGD

vqSGD: Vector Quantized Stochastic Gradient Descent

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning

Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients

Double Quantization for Communication-Efficient Distributed Optimization

Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

AC-SGD: Adaptively Compressed SGD for Communication-Efficient Distributed Learning

Distributed Subgradient Method with Random Quantization and Flexible Weights: Convergence Analysis.

Distributed Online Optimization Under Dynamic Adaptive Quantization

Log-Scale Quantization in Distributed First-Order Methods: Gradient-based Learning from Distributed Data

A Distributed Training Algorithm of Generative Adversarial Networks with Quantized Gradients

Communication-Censored Distributed Stochastic Gradient Descent

Distributed Online Quantile Regression over Networks with Quantized Communication.

Quantized Distributed Gradient Tracking Algorithm with Linear Convergence in Directed Networks