Abstract: We propose and analyze several stochastic gradient algorithms for finding stationary points or local minimum in nonconvex, possibly with nonsmooth regularizer, finite-sum and online optimization problems. First, we propose a simple proximal stochastic gradient algorithm based on variance reduction called ProxSVRG+. We provide a clean and tight analysis of ProxSVRG+, which shows that it outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, hence solves an open problem proposed in Reddi et al. (2016b). Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG (Reddi et al., 2016b) and extends to the online setting by avoiding full gradient computations. Then, we further propose an optimal algorithm, called SSRGD, based on SARAH (Nguyen et al., 2017) and show that SSRGD further improves the gradient complexity of ProxSVRG+ and achieves the optimal upper bound, matching the known lower bound of (Fang et al., 2018; Li et al., 2021). Moreover, we show that both ProxSVRG+ and SSRGD enjoy automatic adaptation with local structure of the objective function such as the Polyak-\L{}ojasiewicz (PL) condition for nonconvex functions in the finite-sum case, i.e., we prove that both of them can automatically switch to faster global linear convergence without any restart performed in prior work ProxSVRG (Reddi et al., 2016b). Finally, we focus on the more challenging problem of finding an $(\epsilon, \delta)$-local minimum instead of just finding an $\epsilon$-approximate (first-order) stationary point (which may be some bad unstable saddle points). We show that SSRGD can find an $(\epsilon, \delta)$-local minimum by simply adding some random perturbations. Our algorithm is almost as simple as its counterpart for finding stationary points, and achieves similar optimal rates.

Subgradient ellipsoid method for nonsmooth convex problems

A descent subgradient method using Mifflin's line search for nonsmooth nonconvex optimization

Subgradient Projection Algorithms for Convex Feasibility on Riemannian Manifolds with Lower Bounded Curvatures

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

A descent subgradient method using Mifflin line search for nonsmooth nonconvex optimization

An effective subgradient algorithm via Mifflin's line search for nonsmooth nonconvex multiobjective optimization

On Some Versions of Subspace Optimization Methods with Inexact Gradient Information

Subgradient-based Neural Networks for Nonsmooth Nonconvex Optimization Problems

Revisiting Subgradient Method: Complexity and Convergence Beyond Lipschitz Continuity

Computing subgradients of convex relaxations for solutions of parametric ordinary differential equations

Convex Bi-Level Optimization Problems with Non-smooth Outer Objective Function

An Infeasible-Point Subgradient Method Using Adaptive Approximate Projections

Accelerated inertial subgradient extragradient algorithms with non-monotonic step sizes for equilibrium problems and fixed point problems

Yet another fast variant of Newton’s method for nonconvex optimization

New Generalized Derivatives for Solving Variational Inequalities Using the Nonsmooth Newton Methods

On a globally convergent semismooth* Newton method in nonsmooth nonconvex optimzation

Horoballs and the subgradient method

Nonsmooth projection-free optimization with functional constraints

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Subgradient sampling for nonsmooth nonconvex minimization

Analyzing the speed of convergence in nonsmooth optimization via the Goldstein subdifferential with application to descent methods