Abstract:Communication compression is a common technique in distributed optimization that can alleviate communication overhead by transmitting compressed gradients and model parameters. However, compression can introduce information distortion, which slows down convergence and incurs more communication rounds to achieve desired solutions. Given the trade-off between lower per-round communication costs and additional rounds of communication, it is unclear whether communication compression reduces the total communication cost. This paper explores the conditions under which unbiased compression, a widely used form of compression, can reduce the total communication cost, as well as the extent to which it can do so. To this end, we present the first theoretical formulation for characterizing the total communication cost in distributed optimization with communication compression. We demonstrate that unbiased compression alone does not necessarily save the total communication cost, but this outcome can be achieved if the compressors used by all workers are further assumed independent. We establish lower bounds on the communication rounds required by algorithms using independent unbiased compressors to minimize smooth convex functions and show that these lower bounds are tight by refining the analysis for ADIANA. Our results reveal that using independent unbiased compression can reduce the total communication cost by a factor of up to $\Theta(\sqrt{\min\{n, \kappa\}})$ when all local smoothness constants are constrained by a common upper bound, where $n$ is the number of workers and $\kappa$ is the condition number of the functions being minimized. These theoretical findings are supported by experimental results.

Communication Compression for Distributed Nonconvex Optimization

Compressed Gradient Tracking Algorithms for Distributed Nonconvex Optimization

Differentially Private and Communication-Efficient Distributed Nonconvex Optimization Algorithms

Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression

Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?

Spatio-Temporal Communication Compression for Distributed Prime-Dual Optimization

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

A Communication-Efficient Stochastic Gradient Descent Algorithm for Distributed Nonconvex Optimization

An Initialization-Free Continuous-Time Distributed Algorithm for Convex Optimization Problem

Spatio-Temporal Communication Compression in Distributed Prime-Dual Flows

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization

Linear Convergence of First- and Zeroth-Order Primal-Dual Algorithms for Distributed Nonconvex Optimization

Compressed Differentially Private Distributed Optimization with Linear Convergence

Communication-Efficient Network-Distributed Optimization with Differential-Coded Compressors

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Quantized Distributed Nonconvex Optimization Algorithms with Linear Convergence under the Polyak--$Ł$ojasiewicz Condition

Real Acceleration of Communication Process in Distributed Algorithms with Compression

Linearly Convergent Distributed Optimization Methods with Compressed Innovation

A Unified Algorithmic Framework for Distributed Composite Optimization.