Abstract: We propose and analyze several stochastic gradient algorithms for finding stationary points or local minimum in nonconvex, possibly with nonsmooth regularizer, finite-sum and online optimization problems. First, we propose a simple proximal stochastic gradient algorithm based on variance reduction called ProxSVRG+. We provide a clean and tight analysis of ProxSVRG+, which shows that it outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, hence solves an open problem proposed in Reddi et al. (2016b). Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG (Reddi et al., 2016b) and extends to the online setting by avoiding full gradient computations. Then, we further propose an optimal algorithm, called SSRGD, based on SARAH (Nguyen et al., 2017) and show that SSRGD further improves the gradient complexity of ProxSVRG+ and achieves the optimal upper bound, matching the known lower bound of (Fang et al., 2018; Li et al., 2021). Moreover, we show that both ProxSVRG+ and SSRGD enjoy automatic adaptation with local structure of the objective function such as the Polyak-\L{}ojasiewicz (PL) condition for nonconvex functions in the finite-sum case, i.e., we prove that both of them can automatically switch to faster global linear convergence without any restart performed in prior work ProxSVRG (Reddi et al., 2016b). Finally, we focus on the more challenging problem of finding an $(\epsilon, \delta)$-local minimum instead of just finding an $\epsilon$-approximate (first-order) stationary point (which may be some bad unstable saddle points). We show that SSRGD can find an $(\epsilon, \delta)$-local minimum by simply adding some random perturbations. Our algorithm is almost as simple as its counterpart for finding stationary points, and achieves similar optimal rates.

Sharp Analysis of Simple Restarted Stochastic Gradient Methods for Min-Max Optimization

Stability and Generalization of Stochastic Gradient Methods for Minimax Problems

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Stochastic Primal-Dual Algorithms with Faster Convergence Than O(1/√T) for Problems Without Bilinear Structure

Sharper Analysis for Minibatch Stochastic Proximal Point Methods: Stability, Smoothness, and Deviation.

Sharp Analysis for Nonconvex SGD Escaping from Saddle Points

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Towards Sharper Risk Bounds for Minimax Problems

A Stochastic GDA Method With Backtracking For Solving Nonconvex (Strongly) Concave Minimax Problems

Stochastic Primal-Dual Algorithms with Faster Convergence than $O(1/\sqrt{T})$ for Problems without Bilinear Structure

Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima

Diffusion Stochastic Optimization for Min-Max Problems

Decentralized gradient descent maximization method for composite nonconvex strongly-concave minimax problems

Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learning

How to escape sharp minima with random perturbations

A Single-Loop Accelerated Extra-Gradient Difference Algorithm with Improved Complexity Bounds for Constrained Minimax Optimization.

On Stochastic Primal-Dual Hybrid Gradient Approach for Compositely Regularized Minimization.

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Two-Timescale Gradient Descent Ascent Algorithms for Nonconvex Minimax Optimization

Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy