Abstract: We propose and analyze several stochastic gradient algorithms for finding stationary points or local minimum in nonconvex, possibly with nonsmooth regularizer, finite-sum and online optimization problems. First, we propose a simple proximal stochastic gradient algorithm based on variance reduction called ProxSVRG+. We provide a clean and tight analysis of ProxSVRG+, which shows that it outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, hence solves an open problem proposed in Reddi et al. (2016b). Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG (Reddi et al., 2016b) and extends to the online setting by avoiding full gradient computations. Then, we further propose an optimal algorithm, called SSRGD, based on SARAH (Nguyen et al., 2017) and show that SSRGD further improves the gradient complexity of ProxSVRG+ and achieves the optimal upper bound, matching the known lower bound of (Fang et al., 2018; Li et al., 2021). Moreover, we show that both ProxSVRG+ and SSRGD enjoy automatic adaptation with local structure of the objective function such as the Polyak-\L{}ojasiewicz (PL) condition for nonconvex functions in the finite-sum case, i.e., we prove that both of them can automatically switch to faster global linear convergence without any restart performed in prior work ProxSVRG (Reddi et al., 2016b). Finally, we focus on the more challenging problem of finding an $(\epsilon, \delta)$-local minimum instead of just finding an $\epsilon$-approximate (first-order) stationary point (which may be some bad unstable saddle points). We show that SSRGD can find an $(\epsilon, \delta)$-local minimum by simply adding some random perturbations. Our algorithm is almost as simple as its counterpart for finding stationary points, and achieves similar optimal rates.

Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition

Accelerated Stochastic Subgradient Methods under Local Error Bound Condition

Convergence Analysis of Accelerated Stochastic Gradient Descent under the Growth Condition

Accelerated Convex Optimization with Stochastic Gradients: Generalizing the Strong-Growth Condition

Accelerated gradient methods for nonconvex nonlinear and stochastic programming

Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence

Accelerated Almost-Sure Convergence Rates for Nonconvex Stochastic Gradient Descent using Stochastic Learning Rates

An Accelerated Decentralized Stochastic Optimization Algorithm with Inexact Model

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization

Local Convergence Properties of SAGA/Prox-SVRG and Acceleration

Local Convergence Properties of SAGA/Prox-SVRG and Acceleration.

Accelerated stochastic approximation with state-dependent noise

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Stochastic smoothing accelerated gradient method for nonsmooth convex composite optimization

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Predictive Local Smoothness for Stochastic Gradient Methods

Accelerated first-order methods for convex optimization with locally Lipschitz continuous gradient

Accelerated Gradient Methods with Gradient Restart: Global Linear Convergence

Optimal Adaptive and Accelerated Stochastic Gradient Descent