Abstract: We propose and analyze several stochastic gradient algorithms for finding stationary points or local minimum in nonconvex, possibly with nonsmooth regularizer, finite-sum and online optimization problems. First, we propose a simple proximal stochastic gradient algorithm based on variance reduction called ProxSVRG+. We provide a clean and tight analysis of ProxSVRG+, which shows that it outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, hence solves an open problem proposed in Reddi et al. (2016b). Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG (Reddi et al., 2016b) and extends to the online setting by avoiding full gradient computations. Then, we further propose an optimal algorithm, called SSRGD, based on SARAH (Nguyen et al., 2017) and show that SSRGD further improves the gradient complexity of ProxSVRG+ and achieves the optimal upper bound, matching the known lower bound of (Fang et al., 2018; Li et al., 2021). Moreover, we show that both ProxSVRG+ and SSRGD enjoy automatic adaptation with local structure of the objective function such as the Polyak-\L{}ojasiewicz (PL) condition for nonconvex functions in the finite-sum case, i.e., we prove that both of them can automatically switch to faster global linear convergence without any restart performed in prior work ProxSVRG (Reddi et al., 2016b). Finally, we focus on the more challenging problem of finding an $(\epsilon, \delta)$-local minimum instead of just finding an $\epsilon$-approximate (first-order) stationary point (which may be some bad unstable saddle points). We show that SSRGD can find an $(\epsilon, \delta)$-local minimum by simply adding some random perturbations. Our algorithm is almost as simple as its counterpart for finding stationary points, and achieves similar optimal rates.

Complexity of Finding Stationary Points of Nonconvex Nonsmooth Functions.

Sample Complexity of Policy Gradient Finding Second-Order Stationary Points

The Computational Complexity of Finding Stationary Points in Non-Convex Optimization

Hybrid Algorithms for Finding a D-Stationary Point of a Class of Structured Nonsmooth DC Minimization

Finding Second-Order Stationary Points in Nonconvex-Strongly-Concave Minimax Optimization

Near-Optimal Fully First-Order Algorithms for Finding Stationary Points in Bilevel Optimization

Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization

Complexity Lower Bounds for Nonconvex-Strongly-Concave Min-Max Optimization.

Zeroth-Order Methods for Constrained Nonconvex Nonsmooth Stochastic Optimization

High-Probability Bound for Non-Smooth Non-Convex Stochastic Optimization with Heavy Tails

Goldstein Stationarity in Lipschitz Constrained Optimization

Online Optimization Perspective on First-Order and Zero-Order Decentralized Nonsmooth Nonconvex Stochastic Optimization

Finding Second-Order Stationary Point for Nonconvex-Strongly-Concave Minimax Problem

An Algorithm with Optimal Dimension-Dependence for Zero-Order Nonsmooth Nonconvex Stochastic Optimization

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

First-Order Methods for Nonsmooth Nonconvex Functional Constrained Optimization with or without Slater Points

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

On Convergence of Training Loss Without Reaching Stationary Points

Stationarity in nonsmooth optimization between geometrical motivation and topological relevance

Complexity-optimal and parameter-free first-order methods for finding stationary points of composite optimization problems