Abstract:Gradient quantization is widely adopted to mitigate communication costs in distributed learning systems. Existing gradient quantization algorithms often rely on design heuristics and/or empirical evidence to tune the quantization strategy for different learning problems. To the best of our knowledge, there is no theoretical framework characterizing the trade-off between communication cost and model accuracy under dynamic gradient quantization strategies. This paper addresses this issue by proposing a novel dynamic quantized SGD (DQSGD) framework, which enables us to optimize the quantization strategy for each gradient descent step by exploring the trade-off between communication cost and modeling error. In particular, we derive an upper bound, tight in some cases, of the modeling error for arbitrary dynamic quantization strategy. By minimizing this upper bound, we obtain an enhanced quantization algorithm with significantly improved modeling error under given communication overhead constraints. Besides, we show that our quantization scheme achieves a strengthened communication cost and model accuracy trade-off in a wide range of optimization models. Finally, through extensive experiments on large-scale computer vision and natural language processing tasks on CIFAR-10, CIFAR-100, and AG-News datasets, respectively. we demonstrate that our quantization scheme significantly outperforms the state-of-the-art gradient quantization methods in terms of communication costs.

Model transformation based distributed stochastic gradient algorithm for multivariate output-error systems

Distributed identification based partially-coupled recursive generalized extended least squares algorithm for multivariate input–output-error systems with colored noises from observation data

Decomposition‐based maximum likelihood gradient iterative algorithm for multivariate systems with colored noise

Hierarchical Gradient Based Iterative Parameter Estimation Algorithm for Multivariable Output Error Moving Average Systems

Robust Gradient Estimation Algorithm for a Stochastic System with Colored Noise

Hierarchical estimation methods based on the penalty term for controlled autoregressive systems with colored noises

Parameter Estimation for Multivariate Nonlinear Stochastic Differential Equation Models: A Comparison Study

DisSAGD: A Distributed Parameter Update Scheme Based on Variance Reduction

A novel multivariate grey system model with conformable fractional derivative and its applications

Differential Equations for Modeling Asynchronous Algorithms

Quantized Adam with Error Feedback

DQSGD: DYNAMIC QUANTIZED STOCHASTIC GRADIENT DESCENT FOR COMMUNICATION-EFFICIENT DISTRIBUTED LEARNING

Data Filtering-Based Multi-innovation Stochastic Gradient Algorithm for Nonlinear Output Error Autoregressive Systems

A Stein Gradient Descent Approach for Doubly Intractable Distributions

Hierarchical Stochastic Gradient Algorithm and its Performance Analysis for a Class of Bilinear-in-Parameter Systems

Online Parameter Optimization Scheme for the Kernel Function-Based Mixture Models Disturbed by Colored Noises

Maximum likelihood based multi‐innovation stochastic gradient identification algorithms for bilinear stochastic systems with ARMA noise

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations

Weighted multi‐innovation parameter estimation for a time‐varying Volterra–Hammerstein system with colored noise

A modified stochastic gradient based parameter estimation algorithm for dual-rate sampled-data systems