Abstract:Variations of stochastic gradient decedent (SGD) methods are at the core of training deep neural network models. However, in distributed deep learning, where multiple computing devices and data segments are employed in the training process, the performance of SGD can be significantly limited by the overhead of gradient communication. Local SGD methods are designed to overcome this bottleneck by averaging individual gradients trained over parallel workers after multiple local iterations. Currently, both for theoretical analyses and for practical applications, most studies employ periodic synchronization scheme by default, while few of them focus on the aperiodic schemes to obtain better performance models with limited computation and communication overhead. In this paper, we investigate local SGD with an arbitrary synchronization scheme to answer two questions: (1) Is the periodic synchronization scheme best? (2) If not, what is the optimal one? First, for any synchronization scheme, we derive the performance boundary with fixed overhead, and formulate the performance optimization under given computation and communication constraints. Then we find a succinct property of the optimal scheme that the local iteration number decreases as training continues, which indicates the periodic one is suboptimal. Furthermore, with some reasonable approximations, we obtain an explicit form of the optimal scheme and propose Aperiodic Local SGD (ALSGD) as an improved substitute for local SGD without any overhead increment. Our experiments also confirm that with the same computation and communication overhead, ALSGD outperforms local SGD in performance, especially for heterogeneous data.

Statistical Estimation and Online Inference via Local SGD.

Statistical Estimation and Inference via Local SGD in Federated Learning

On the Convergence of Communication-Efficient Local SGD for Federated Learning

Communication-Efficient Local Decentralized SGD Methods

On the Convergence of Local Descent Methods in Federated Learning

Decentralized Federated Learning: Balancing Communication and Computing Costs

STL-SGD: Speeding Up Local SGD with Stagewise Communication Period

Federated Stochastic Gradient Langevin Dynamics

Resilient Two-Time-Scale Local Stochastic Gradient Descent for Byzantine Federated Learning

The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates

General Decentralized Federated Learning for Communication-Computation Tradeoff.

Communication Efficient Decentralized Training with Multiple Local Updates.

SLowcal-SGD: Slow Query Points Improve Local-SGD for Stochastic Convex Optimization

DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updates

Why (and When) does Local SGD Generalize Better than SGD?

Accelerating local SGD for non-IID data using variance reduction

A Statistical Online Inference Approach in Averaged Stochastic Approximation

Aperiodic Local SGD: Beyond Local SGD.

Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regression

Decentralized Federated Learning with Unreliable Communications