Abstract:Many modern large-scale machine learning problems benefit from decentralized and stochastic optimization. Recent works have shown that utilizing both decentralized computing and local stochastic gradient estimates can outperform state-of-the-art centralized algorithms, in applications involving highly non-convex problems, such as training deep neural networks. In this work, we propose a decentralized stochastic algorithm to deal with certain smooth non-convex problems where there are $m$ nodes in the system, and each node has a large number of samples (denoted as $n$). Differently from the majority of the existing decentralized learning algorithms for either stochastic or finite-sum problems, our focus is given to both reducing the total communication rounds among the nodes, while accessing the minimum number of local data samples. In particular, we propose an algorithm named D-GET (decentralized gradient estimation and tracking), which jointly performs decentralized gradient estimation (which estimates the local gradient using a subset of local samples) and gradient tracking (which tracks the global full gradient using local estimates). We show that, to achieve certain $\epsilon$ stationary solution of the deterministic finite sum problem, the proposed algorithm achieves an $\mathcal{O}(mn^{1/2}\epsilon^{-1})$ sample complexity and an $\mathcal{O}(\epsilon^{-1})$ communication complexity. These bounds significantly improve upon the best existing bounds of $\mathcal{O}(mn\epsilon^{-1})$ and $\mathcal{O}(\epsilon^{-1})$, respectively. Similarly, for online problems, the proposed method achieves an $\mathcal{O}(m \epsilon^{-3/2})$ sample complexity and an $\mathcal{O}(\epsilon^{-1})$ communication complexity, while the best existing bounds are $\mathcal{O}(m\epsilon^{-2})$ and $\mathcal{O}(\epsilon^{-2})$, respectively.

The Incremental Subgradient Methods on Distributed Estimations In-Network

Diffusion Information Theoretic Learning for Distributed Estimation over Network

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Distributed Sparse Recursive Least-Squares over Networks

Distributed alternating gradient descent for convex semi-infinite programs over a network

Distributed estimation over complex networks

Distributed Stochastic Algorithm for Global Optimization in Networked System

Estimation Network Design framework for efficient distributed optimization

Distributed Jointly Sparse Multitask Learning over Networks

A Decentralized Continuous Estimation Of Distribution Algorithm For Networked Systems

Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex Optimization

Communication-efficient Distributed Newton-like Optimization with Gradients and M-estimators

Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates

Randomized Incremental Least Squares for Distributed Estimation over Sensor Networks

Distributed Stochastic Optimization With Unbounded Subgradients Over Randomly Time-Varying Networks

Distributed Experimental Design Networks

A Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent

Distributed quasi-monotone subgradient algorithm for nonsmooth convex optimization over directed graphs

Asymptotic Network Independence in Distributed Stochastic Optimization for Machine Learning

Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approach

Decentralized stochastic subgradient projection optimization algorithms over random networks