Abstract:Communication-constrained algorithms for decentralized learning and optimization rely on local updates coupled with the exchange of compressed signals. In this context, differential quantization is an effective technique to mitigate the negative impact of compression by leveraging correlations between successive iterates. In addition, the use of error feedback, which consists of incorporating the compression error into subsequent steps, is a powerful mechanism to compensate for the bias caused by the compression. Under error feedback, performance guarantees in the literature have so far focused on algorithms employing a fusion center or a special class of contractive compressors that cannot be implemented with a finite number of bits. In this work, we propose a new decentralized communication-efficient learning approach that blends differential quantization with error feedback. The approach is specifically tailored for decentralized learning problems where agents have individual risk functions to minimize subject to subspace constraints that require the minimizers across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus or single-task optimization as special cases, and allows for more general task relatedness models such as multitask smoothness and coupled optimization. We show that, under some general conditions on the compression noise, and for sufficiently small step-sizes $\mu$, the resulting communication-efficient strategy is stable both in terms of mean-square error and average bit rate: by reducing $\mu$, it is possible to keep the estimation errors small (on the order of $\mu$) without increasing indefinitely the bit rate as $\mu\rightarrow 0$. The results establish that, in the small step-size regime and with a finite number of bits, it is possible to attain the performance achievable in the absence of compression.

DEED: A General Quantization Scheme for Communication Efficiency in Bits

Double Quantization for Communication-Efficient Distributed Optimization

DQSGD: DYNAMIC QUANTIZED STOCHASTIC GRADIENT DESCENT FOR COMMUNICATION-EFFICIENT DISTRIBUTED LEARNING

vqSGD: Vector Quantized Stochastic Gradient Descent

DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

Distributed Optimization with Finite Bit Adaptive Quantization for Efficient Communication and Precision Enhancement

Finite-Bit Quantization for Distributed Algorithms With Linear Convergence

Communication-Efficient Design for Quantized Decentralized Federated Learning

Distributed Online Optimization Under Dynamic Adaptive Quantization

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

Quantized Epoch-SGD for Communication-Efficient Distributed Learning.

Cloud-Edge Inference under Communication Constraints: Data Quantization and Early Exit.

Distributed Online Quantile Regression over Networks with Quantized Communication.

Exploring Fast Algorithms for Composite Optimization with Serial and Asynchronous Realizations.

Differential error feedback for communication-efficient decentralized learning

Distributed Optimization via Gradient Descent with Event-Triggered Zooming over Quantized Communication

Quantization Avoids Saddle Points in Distributed Optimization

Exploring Fast Algorithms for Composite Optimization with Serial and Asynchronous Realizations

Killing Two Birds with One Stone: Quantization Achieves Privacy in Distributed Learning