Abstract:Communication is a crucial phase in the context of distributed training. Because parameter server (PS) frequently experiences network congestion, recent studies have found that training paradigms without a centralized server outperform the traditional server-based paradigms in terms of communication efficiency. However, with the increasing growth of model sizes, these server-free paradigms are also confronted with substantial communication overhead that seriously deteriorates the performance of distributed training. In this paper, we focus on communication efficiency of two serverless paradigms, i.e., Ring All-Reduce (RAR) and gossip, by proposing the Quantized Parallel Restarted Stochastic Gradient Descent (QPRSGD), an algorithm that allows multiple local SGD updates before a global synchronization, in synergy with the quantization to significantly reduce the communication overhead. We establish the bound of accumulative errors according to the synchronization mode and the network topology, which is essential to ensure the convergence property. Under both aggregation paradigms, the algorithm achieves the linear speedup property with respect to the number of local updates as well as the number of workers. Remarkably, the proposed algorithm achieves a convergence rate $O(1/\sqrt{NK^2M})$ under the gossip paradigm and outperforms all existing compression methods, where $N$ is the times of global synchronizations, and $K$ is the number of local updates, while $M$ is the number of nodes. An empirical study on various machine learning models demonstrates that the communication overhead is reduced by 90\%, and the convergence speed is boosted by up to 18.6 times, in a low bandwidth network, in comparison with Parallel SGD.

Communication Efficient Decentralized Training with Multiple Local Updates

Communication Efficient Decentralized Training with Multiple Local Updates.

Communication-Efficient Local Decentralized SGD Methods

STL-SGD: Speeding Up Local SGD with Stagewise Communication Period

The Effectiveness of Local Updates for Decentralized Learning under Data Heterogeneity

On the Convergence of Quantized Parallel Restarted SGD for Central Server Free Distributed Training

Communication-Censored Distributed Stochastic Gradient Descent

A Graph Neural Network Based Decentralized Learning Scheme

Communication-Efficient Local SGD with Age-Based Worker Selection

Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates

Aperiodic Local SGD: Beyond Local SGD.

Communication-Efficient Topologies for Decentralized Learning with O(1) Consensus Rate

Decentralized Federated Learning with Unreliable Communications

Can We Learn Communication-Efficient Optimizers?

DSGD-CECA: Decentralized SGD with Communication-Optimal Exact Consensus Algorithm

Variance Reduced Local SGD with Lower Communication Complexity

Decentralized Learning with Lazy and Approximate Dual Gradients

A Communication-Efficient Stochastic Gradient Descent Algorithm for Distributed Nonconvex Optimization

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

General Decentralized Federated Learning for Communication-Computation Tradeoff.

Decentralized Federated Learning: Balancing Communication and Computing Costs