Abstract:Stochastic Gradient (SG) is the defacto iterative technique to solve stochastic optimization (SO) problems with a smooth (non-convex) objective $f$ and a stochastic first-order oracle. SG's attractiveness is due in part to its simplicity of executing a single step along the negative subsampled gradient direction to update the incumbent iterate. In this paper, we question SG's choice of executing a single step as opposed to multiple steps between subsample updates. Our investigation leads naturally to generalizing SG into Retrospective Approximation (RA) where, during each iteration, a "deterministic solver" executes possibly multiple steps on a subsampled deterministic problem and stops when further solving is deemed unnecessary from the standpoint of statistical efficiency. RA thus rigorizes what is appealing for implementation -- during each iteration, "plug in" a solver, e.g., L-BFGS line search or Newton-CG, as is, and solve only to the extent necessary. We develop a complete theory using relative error of the observed gradients as the principal object, demonstrating that almost sure and $L_1$ consistency of RA are preserved under especially weak conditions when sample sizes are increased at appropriate rates. We also characterize the iteration and oracle complexity (for linear and sub-linear solvers) of RA, and identify a practical termination criterion leading to optimal complexity rates. To subsume non-convex $f$, we present a certain "random central limit theorem" that incorporates the effect of curvature across all first-order critical points, demonstrating that the asymptotic behavior is described by a certain mixture of normals. The message from our numerical experiments is that the ability of RA to incorporate existing second-order deterministic solvers in a strategic manner might be important from the standpoint of dispensing with hyper-parameter tuning.

Truncated Cauchy random perturbations for smoothed functional-based stochastic optimization

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

A Retrospective Approximation Approach for Smooth Stochastic Optimization

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation

Derivatives of Stochastic Gradient Descent in parametric optimization

Accelerated stochastic approximation with state-dependent noise

High-Probability Bound for Non-Smooth Non-Convex Stochastic Optimization with Heavy Tails

A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness

How to escape sharp minima with random perturbations

A stochastic use of the Kurdyka-Lojasiewicz property: Investigation of optimization algorithms behaviours in a non-convex differentiable framework

Accelerated zero-order SGD under high-order smoothness and overparameterized regime

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

A relaxation-based probabilistic approach for PDE-constrained optimization under uncertainty with pointwise state constraints

Optimal sampling for stochastic and natural gradient descent

Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descent

Almost Sure Convergence of Randomised‐difference Descent Algorithm for Stochastic Convex Optimisation

Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization