Abstract:We are investigating the distributed optimization problem, where a network of nodes works together to minimize a global objective that is a finite sum of their stored local functions. Since nodes exchange optimization parameters through the wireless network, large-scale training models can create communication bottlenecks, resulting in slower training times. To address this issue, CHOCO-SGD was proposed, which allows compressing information with arbitrary precision without reducing the convergence rate for strongly convex objective functions. Nevertheless, most convex functions are not strongly convex (such as logistic regression or Lasso), which raises the question of whether this algorithm can be applied to non-strongly convex functions. In this paper, we provide the first theoretical analysis of the convergence rate of CHOCO-SGD on non-strongly convex objectives. We derive a sufficient condition, which limits the fidelity of compression, to guarantee convergence. Moreover, our analysis demonstrates that within the fidelity threshold, this algorithm can significantly reduce transmission burden while maintaining the same convergence rate order as its no-compression equivalent. Numerical experiments further validate the theoretical findings by demonstrating that CHOCO-SGD improves communication efficiency and keeps the same convergence rate order simultaneously. And experiments also show that the algorithm fails to converge with low compression fidelity and in time-varying topologies. Overall, our study offers valuable insights into the potential applicability of CHOCO-SGD for non-strongly convex objectives. Additionally, we provide practical guidelines for researchers seeking to utilize this algorithm in real-world scenarios.

Innovation Compression for Communication-efficient Distributed Optimization with Linear Convergence

Linearly Convergent Distributed Optimization Methods with Compressed Innovation

Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression

A Communication-Efficient Distributed Optimization Algorithm with Linear Convergence over Directed Graphs

Communication Compression for Distributed Nonconvex Optimization

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Distributed Constrained Optimization with Linear Convergence Rate

Quantized Distributed Nonconvex Optimization Algorithms with Linear Convergence under the Polyak--$Ł$ojasiewicz Condition

Communication-Efficient Network-Distributed Optimization with Differential-Coded Compressors

A Unified Algorithmic Framework for Distributed Composite Optimization.

Linear Convergent Distributed Nash Equilibrium Seeking with Compression

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Distributed Stochastic Optimization with Compression for Non-Strongly Convex Objectives

Compressed Differentially Private Distributed Optimization with Linear Convergence

Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?

Distributed Solvers for Network Linear Equations with Scalarized Compression

A Linearly Convergent Robust Compressed Push-Pull Method for Decentralized Optimization

A Compressed Gradient Tracking Method for Decentralized Optimization With Linear Convergence

A Communication-efficient Linearly Convergent Algorithm with Variance Reduction for Distributed Stochastic Optimization

Differentially Private and Communication-Efficient Distributed Nonconvex Optimization Algorithms

Linear convergence of a dual optimization formulation for distributed optimization on directed graphs with unreliable communications