Abstract: We propose and analyze several stochastic gradient algorithms for finding stationary points or local minimum in nonconvex, possibly with nonsmooth regularizer, finite-sum and online optimization problems. First, we propose a simple proximal stochastic gradient algorithm based on variance reduction called ProxSVRG+. We provide a clean and tight analysis of ProxSVRG+, which shows that it outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, hence solves an open problem proposed in Reddi et al. (2016b). Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG (Reddi et al., 2016b) and extends to the online setting by avoiding full gradient computations. Then, we further propose an optimal algorithm, called SSRGD, based on SARAH (Nguyen et al., 2017) and show that SSRGD further improves the gradient complexity of ProxSVRG+ and achieves the optimal upper bound, matching the known lower bound of (Fang et al., 2018; Li et al., 2021). Moreover, we show that both ProxSVRG+ and SSRGD enjoy automatic adaptation with local structure of the objective function such as the Polyak-\L{}ojasiewicz (PL) condition for nonconvex functions in the finite-sum case, i.e., we prove that both of them can automatically switch to faster global linear convergence without any restart performed in prior work ProxSVRG (Reddi et al., 2016b). Finally, we focus on the more challenging problem of finding an $(\epsilon, \delta)$-local minimum instead of just finding an $\epsilon$-approximate (first-order) stationary point (which may be some bad unstable saddle points). We show that SSRGD can find an $(\epsilon, \delta)$-local minimum by simply adding some random perturbations. Our algorithm is almost as simple as its counterpart for finding stationary points, and achieves similar optimal rates.

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient

Online Successive Convex Approximation for Two-Stage Stochastic Nonconvex Optimization

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Trust-Region Sequential Quadratic Programming for Stochastic Optimization with Random Models

A simple and practical adaptive trust-region method

A TRUST-REGION METHOD FOR NONSMOOTH NONCONVEX OPTIMIZATION

On inexact stochastic splitting methods for a class of nonconvex composite optimization problems with relative error

Stochastic Optimization for Non-convex Inf-Projection Problems

Stochastic Newton Proximal Extragradient Method

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

A stochastic use of the Kurdyka-Lojasiewicz property: Investigation of optimization algorithms behaviours in a non-convex differentiable framework

An Improved Non-Monotonic Adaptive Trust Region Algorithm for Unconstrained Optimization

TRAFS: A Nonsmooth Convex Optimization Algorithm with $\mathcal{O}\left(\frac{1}ε\right)$ Iteration Complexity

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the O(1/T) Convergence Rate.

Fully Stochastic Trust-Region Sequential Quadratic Programming for Equality-Constrained Optimization Problems

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Asymptotic Optimality in Stochastic Optimization

Stochastic Approximation Proximal Subgradient Method for Stochastic Convex-Concave Minimax Optimization

Stochastic Optimization under Hidden Convexity