Abstract:Communication-constrained algorithms for decentralized learning and optimization rely on local updates coupled with the exchange of compressed signals. In this context, differential quantization is an effective technique to mitigate the negative impact of compression by leveraging correlations between successive iterates. In addition, the use of error feedback, which consists of incorporating the compression error into subsequent steps, is a powerful mechanism to compensate for the bias caused by the compression. Under error feedback, performance guarantees in the literature have so far focused on algorithms employing a fusion center or a special class of contractive compressors that cannot be implemented with a finite number of bits. In this work, we propose a new decentralized communication-efficient learning approach that blends differential quantization with error feedback. The approach is specifically tailored for decentralized learning problems where agents have individual risk functions to minimize subject to subspace constraints that require the minimizers across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus or single-task optimization as special cases, and allows for more general task relatedness models such as multitask smoothness and coupled optimization. We show that, under some general conditions on the compression noise, and for sufficiently small step-sizes $\mu$, the resulting communication-efficient strategy is stable both in terms of mean-square error and average bit rate: by reducing $\mu$, it is possible to keep the estimation errors small (on the order of $\mu$) without increasing indefinitely the bit rate as $\mu\rightarrow 0$. The results establish that, in the small step-size regime and with a finite number of bits, it is possible to attain the performance achievable in the absence of compression.

Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Rates

Byzantine-Robust Distributed Learning with Compression.

Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filtering

Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates

Byzantine-Robust Compressed and Momentum-based Variance Reduction in Federated Learning

Byzantine-Robust Aggregation with Gradient Difference Compression and Stochastic Variance Reduction for Federated Learning

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers

C-RSA: Byzantine-robust and Communication-Efficient Distributed Learning in the Non-Convex and Non-Iid Regime.

Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression

Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

Byzantine-Resilient Federated Learning at Edge

Communication-Efficient Byzantine-Resilient Federated Zero-Order Optimization

A Linearly Convergent Robust Compressed Push-Pull Method for Decentralized Optimization

A Robust Compressed Push-Pull Method for Decentralized Nonconvex Optimization

Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences

Dynamic Byzantine-Robust Learning: Adapting to Switching Byzantine Workers

Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression

FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learning

Resilient to byzantine attacks finite-sum optimization over networks

Efficient Byzantine-Resilient Stochastic Gradient Desce

Differential error feedback for communication-efficient decentralized learning