Abstract:This paper describes a novel algorithmic framework to minimize a finite-sum of functions available over a network of nodes. The proposed framework, that we call~\GTVR, is stochastic and decentralized, and thus is particularly suitable for problems where large-scale, potentially private data, cannot be collected or processed at a centralized server. The \GTVR~framework leads to a family of algorithms with two key ingredients: (i) \textit{local variance reduction}, that enables estimating the local batch gradients from arbitrarily drawn samples of local data; and, (ii) \textit{global gradient tracking}, which fuses the gradient information across the nodes. Naturally, combining different variance reduction and gradient tracking techniques leads to different algorithms of interest with valuable practical tradeoffs and design considerations. Our focus in this paper is on two instantiations of the~$\GTVR$ framework, namely~\textbf{\texttt{GT-SAGA}} and~\textbf{\texttt{GT-SVRG}}, that, similar to their centralized counterparts (\SAGA~and~\SVRG), exhibit a compromise between space and time. We show that both~\textbf{\texttt{GT-SAGA}} and~\textbf{\texttt{GT-SVRG}} achieve accelerated linear convergence for smooth and strongly convex problems and further describe the regimes in which they achieve non-asymptotic, network-independent linear convergence rates that are faster with respect to the existing decentralized first-order schemes. Moreover, we show that both algorithms achieve a linear speedup in such regimes, in that, the total number of gradient computations required at each node is reduced by a factor of $1/n$, where $n$ is the number of nodes, compared to their centralized counterparts that process all data at a single node. Extensive simulations illustrate the convergence behavior of the corresponding algorithms.

Local Convergence Properties of SAGA/Prox-SVRG and Acceleration

Local Convergence Properties of SAGA/Prox-SVRG and Acceleration.

Local Convergence Properties of SAGA/Prox-SVRG and Acceleration

Supplementary Material for “Local Convergence Properties of SAGA/Prox SVRG and Acceleration”

Linear Convergence of Variance-Reduced Stochastic Gradient without Strong Convexity

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition

Variance-Reduced Proximal Stochastic Gradient Descent for Non-convex Composite optimization.

Larger is Better: The Effect of Learning Rates Enjoyed by Stochastic Optimization with Progressive Variance Reduction

Sampling and Update Frequencies in Proximal Variance-Reduced Stochastic Gradient Methods

Variance reduction techniques for stochastic proximal point algorithms

Faster Gradient-Free Proximal Stochastic Methods for Nonconvex Nonsmooth Optimization

Asynchronous Stochastic Proximal Methods for Nonconvex Nonsmooth Optimization.

New nonasymptotic convergence rates of stochastic proximal pointalgorithm for convex optimization problems

An Accelerated Variance Reducing Stochastic Method with Douglas-Rachford Splitting.

Variance-Reduced Decentralized Stochastic Optimization with Accelerated Convergence

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

A Simple Linear Convergence Analysis of the Point-SAGA Algorithm

Speeding up Stochastic Proximal Optimization in the High Hessian Dissimilarity Setting

Accelerated Stochastic Subgradient Methods under Local Error Bound Condition

Hybrid Acceleration Scheme for Variance Reduced Stochastic Optimization Algorithms