Abstract: We propose and analyze several stochastic gradient algorithms for finding stationary points or local minimum in nonconvex, possibly with nonsmooth regularizer, finite-sum and online optimization problems. First, we propose a simple proximal stochastic gradient algorithm based on variance reduction called ProxSVRG+. We provide a clean and tight analysis of ProxSVRG+, which shows that it outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, hence solves an open problem proposed in Reddi et al. (2016b). Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG (Reddi et al., 2016b) and extends to the online setting by avoiding full gradient computations. Then, we further propose an optimal algorithm, called SSRGD, based on SARAH (Nguyen et al., 2017) and show that SSRGD further improves the gradient complexity of ProxSVRG+ and achieves the optimal upper bound, matching the known lower bound of (Fang et al., 2018; Li et al., 2021). Moreover, we show that both ProxSVRG+ and SSRGD enjoy automatic adaptation with local structure of the objective function such as the Polyak-\L{}ojasiewicz (PL) condition for nonconvex functions in the finite-sum case, i.e., we prove that both of them can automatically switch to faster global linear convergence without any restart performed in prior work ProxSVRG (Reddi et al., 2016b). Finally, we focus on the more challenging problem of finding an $(\epsilon, \delta)$-local minimum instead of just finding an $\epsilon$-approximate (first-order) stationary point (which may be some bad unstable saddle points). We show that SSRGD can find an $(\epsilon, \delta)$-local minimum by simply adding some random perturbations. Our algorithm is almost as simple as its counterpart for finding stationary points, and achieves similar optimal rates.

Bound Analysis of Natural Gradient Descent in Stochastic Optimization Setting

Understanding Stochastic Natural Gradient Variational Inference

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent

Almost Sure Saddle Avoidance of Stochastic Gradient Methods without the Bounded Gradient Assumption

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Stochastic Gradient Descent in the Viewpoint of Graduated Optimization

Derivatives of Stochastic Gradient Descent in parametric optimization

Riemannian stochastic optimization methods avoid strict saddle points

Convergence Analysis of Stochastic Saddle Point Mirror Descent Algorithm -- A Projected Dynamical View Point

Analysis of Stochastic Gradient Descent in Continuous Time

Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descent

Understanding the unstable convergence of gradient descent.

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects

A KL-based Analysis Framework with Applications to Non-Descent Optimization Methods

Online distributed optimization with stochastic gradients: high probability bound of regrets

High Probability Analysis for Non-Convex Stochastic Optimization with Clipping

High-Probability Bound for Non-Smooth Non-Convex Stochastic Optimization with Heavy Tails

Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergence