Abstract: We propose and analyze several stochastic gradient algorithms for finding stationary points or local minimum in nonconvex, possibly with nonsmooth regularizer, finite-sum and online optimization problems. First, we propose a simple proximal stochastic gradient algorithm based on variance reduction called ProxSVRG+. We provide a clean and tight analysis of ProxSVRG+, which shows that it outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, hence solves an open problem proposed in Reddi et al. (2016b). Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG (Reddi et al., 2016b) and extends to the online setting by avoiding full gradient computations. Then, we further propose an optimal algorithm, called SSRGD, based on SARAH (Nguyen et al., 2017) and show that SSRGD further improves the gradient complexity of ProxSVRG+ and achieves the optimal upper bound, matching the known lower bound of (Fang et al., 2018; Li et al., 2021). Moreover, we show that both ProxSVRG+ and SSRGD enjoy automatic adaptation with local structure of the objective function such as the Polyak-\L{}ojasiewicz (PL) condition for nonconvex functions in the finite-sum case, i.e., we prove that both of them can automatically switch to faster global linear convergence without any restart performed in prior work ProxSVRG (Reddi et al., 2016b). Finally, we focus on the more challenging problem of finding an $(\epsilon, \delta)$-local minimum instead of just finding an $\epsilon$-approximate (first-order) stationary point (which may be some bad unstable saddle points). We show that SSRGD can find an $(\epsilon, \delta)$-local minimum by simply adding some random perturbations. Our algorithm is almost as simple as its counterpart for finding stationary points, and achieves similar optimal rates.

Non-convex finite-sum optimization via scsg methods

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Stochastic smoothing accelerated gradient method for nonsmooth convex composite optimization

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Escaping Saddle Points with Stochastically Controlled Stochastic Gradient Methods

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

Universal Gradient Methods for Stochastic Convex Optimization

Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

Variance-Reduced Proximal Stochastic Gradient Descent for Non-convex Composite optimization.

An optimal method for stochastic composite optimization

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

A Gradient Complexity Analysis for Minimizing the Sum of Strongly Convex Functions with Varying Condition Numbers

Stochastic Coordinate Descent Methods for Regularized Smooth and Nonsmooth Losses.

Predictive Local Smoothness for Stochastic Gradient Methods

Stochastic Gradient Descent in the Viewpoint of Graduated Optimization

Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learning

Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex Optimization

Accelerated stochastic approximation with state-dependent noise

Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Optimization

Painless Stochastic Conjugate Gradient for Large-Scale Machine Learning