Abstract: We propose and analyze several stochastic gradient algorithms for finding stationary points or local minimum in nonconvex, possibly with nonsmooth regularizer, finite-sum and online optimization problems. First, we propose a simple proximal stochastic gradient algorithm based on variance reduction called ProxSVRG+. We provide a clean and tight analysis of ProxSVRG+, which shows that it outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, hence solves an open problem proposed in Reddi et al. (2016b). Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG (Reddi et al., 2016b) and extends to the online setting by avoiding full gradient computations. Then, we further propose an optimal algorithm, called SSRGD, based on SARAH (Nguyen et al., 2017) and show that SSRGD further improves the gradient complexity of ProxSVRG+ and achieves the optimal upper bound, matching the known lower bound of (Fang et al., 2018; Li et al., 2021). Moreover, we show that both ProxSVRG+ and SSRGD enjoy automatic adaptation with local structure of the objective function such as the Polyak-\L{}ojasiewicz (PL) condition for nonconvex functions in the finite-sum case, i.e., we prove that both of them can automatically switch to faster global linear convergence without any restart performed in prior work ProxSVRG (Reddi et al., 2016b). Finally, we focus on the more challenging problem of finding an $(\epsilon, \delta)$-local minimum instead of just finding an $\epsilon$-approximate (first-order) stationary point (which may be some bad unstable saddle points). We show that SSRGD can find an $(\epsilon, \delta)$-local minimum by simply adding some random perturbations. Our algorithm is almost as simple as its counterpart for finding stationary points, and achieves similar optimal rates.

Introduction: Swarm-based gradient descent for non convex optimization

Swarm-Based Optimization with Random Descent

Swarm-based gradient descent meets simulated annealing

An Adaptive Stochastic Dominant Learning Swarm Optimizer for High-Dimensional Optimization

The q-gradient method for global optimization

A Principle for Global Optimization with Gradients

Algebraic and Dynamic Analysis on the Modified Swarm Optimizers

Gradient is All You Need?

A gradient descent akin method for constrained optimization: algorithms and applications

A gentle introduction to gradient-based optimization and variational inequalities for machine learning

An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Population-Based Methods: PARTICLE SWARM OPTIMIZATION -- Development of a General-Purpose Optimizer and Applications

Tropical Gradient Descent

An overview of gradient descent optimization algorithms

Gradient Descent, Stochastic Optimization, and Other Tales

DeepHive: A multi-agent reinforcement learning approach for automated discovery of swarm-based optimization policies

A geometric integration approach to smooth optimisation: Foundations of the discrete gradient method

Gradient Descent: The Ultimate Optimizer

A Flexible Gradient Tracking Algorithmic Framework for Decentralized Optimization

A Communication-Efficient Stochastic Gradient Descent Algorithm for Distributed Nonconvex Optimization