Abstract:We study stochastic decentralized optimization for the problem of training machine learning models with large-scale distributed data. We extend the widely used EXTRA and DIGing methods with variance reduction (VR), and propose two methods: VR-EXTRA and VR-DIGing. The proposed VR-EXTRA requires the time of O((kappa(s)+n) log 1/epsilon ) stochastic gradient evaluations and O((kappa(b) + kappa(c)) log 1/epsilon) communication rounds to reach precision epsilon, which are the best complexities among the non-accelerated gradient-type methods, where kappa(s) and kappa(b) are the stochastic condition number and batch condition number for strongly convex and smooth problems, respectively, kappa(c) is the condition number of the communication network, and n is the sample size on each distributed node. The proposed VR-DIGing has a little higher communication cost of O((kappa(b) + kappa(2)(c)) log 1/epsilon). Our stochastic gradient computation complexities are the same as the ones of single-machine VR methods, such as SAG, SAGA, and SVRG, and our communication complexities keep the same as those of EXTRA and DIGing, respectively. To further speed up the convergence, we also propose the accelerated VR-EXTRA and VR-DIGing with both the optimal O((root n kappa(s)+n) log 1/epsilon) stochastic gradient computation complexity and O(root kappa(b)kappa(c)log 1/epsilon) communication complexity. Our stochastic gradient computation complexity is also the same as the ones of single-machine accelerated VR methods, such as Katyusha, and our communication complexity keeps the same as those of accelerated full batch decentralized methods, such as MSDA. To the best of our knowledge, our accelerated methods are the first to achieve both the optimal stochastic gradient computation complexity and communication complexity in the class of gradient-type methods.

A New Analysis of Variance Reduced Stochastic Proximal Methods for Composite Optimization with Serial and Asynchronous Realizations.

Exploring Fast Algorithms for Composite Optimization with Serial and Asynchronous Realizations

Exploring Fast Algorithms for Composite Optimization with Serial and Asynchronous Realizations.

Double Quantization for Communication-Efficient Distributed Optimization

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization

Asynchronous Stochastic Proximal Methods for Nonconvex Nonsmooth Optimization.

Quantization Avoids Saddle Points in Distributed Optimization

DQSGD: DYNAMIC QUANTIZED STOCHASTIC GRADIENT DESCENT FOR COMMUNICATION-EFFICIENT DISTRIBUTED LEARNING

Distributed Subgradient Method with Random Quantization and Flexible Weights: Convergence Analysis.

Correlated Quantization for Faster Nonconvex Distributed Optimization

Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees

Prox-DBRO-VR: A Unified Analysis on Decentralized Byzantine-Resilient Composite Stochastic Optimization with Variance Reduction and Non-Asymptotic Convergence Rates

Communication-Constrained Distributed Learning: TSI-Aided Asynchronous Optimization with Stale Gradient

Asynchronous Stochastic Composition Optimization with Variance Reduction

Distributed Constrained Optimization With Delayed Subgradient Information Over Time-Varying Network Under Adaptive Quantization

Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients

Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization