Abstract:Gradient-based optimization methods implemented on distributed computing architectures are increasingly used to tackle large-scale machine learning applications. A key bottleneck in such distributed systems is the high communication overhead for exchanging information, such as stochastic gradients, between workers. The inherent causes of this bottleneck are the frequent communication rounds and the full model gradient transmission in every round. In this study, we present SASG, a communication-efficient distributed algorithm that enjoys the advantages of sparse communication and adaptive aggregated stochastic gradients. By dynamically determining the workers who need to communicate through an adaptive aggregation rule and sparsifying the transmitted information, the SASG algorithm reduces both the overhead of communication rounds and the number of communication bits in the distributed system. For the theoretical analysis, we introduce an important auxiliary variable and define a new Lyapunov function to prove that the communication-efficient algorithm is convergent. The convergence result is identical to the sublinear rate of stochastic gradient descent, and our result also reveals that SASG scales well with the number of distributed workers. Finally, experiments on training deep neural networks demonstrate that the proposed algorithm can significantly reduce communication overhead compared to previous methods.

An Accelerated Gossip-Based Distributed Gradient Method

An Accelerated Distributed Gradient Method with Local Memory

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Gossip-based Gradient-Free Method for Multi-Agent Optimization: Constant Step Size Analysis

Augmented Distributed Gradient Methods for Multi-Agent Optimization under Uncoordinated Constant Stepsizes

Gossip-based distributed stochastic mirror descent for constrained optimization

Distributed Subgradient Algorithm for Multi-Agent Optimization with Dynamic Stepsize

A Push-Pull Gradient Method for Distributed Optimization in Networks.

An Accelerated Distributed Stochastic Gradient Method with Momentum

An Accelerated Distributed Method with Inexact Model of Relative Smoothness and Strong Convexity

A Communication-Efficient Stochastic Gradient Descent Algorithm for Distributed Nonconvex Optimization

Gossip Gradient Descent.

A Stochastic Averaging Gradient Algorithm with Multi‐step Communication for Distributed Optimization

GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning

An Accelerated Gradient Tracking Algorithm with Projection Error for Distributed Optimization.

Accelerating Distributed Optimization: A Primal-Dual Perspective on Local Steps

Accelerated Distributed Aggregative Optimization

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

A Distributed Stochastic Optimization Algorithm with Gradient-Tracking and Distributed Heavy-Ball Acceleration

Compressed Distributed Gradient Descent: Communication-Efficient Consensus over Networks

Provably Accelerated Decentralized Gradient Method Over Unbalanced Directed Graphs