Abstract:Stochastic Gradient (SG) is the defacto iterative technique to solve stochastic optimization (SO) problems with a smooth (non-convex) objective $f$ and a stochastic first-order oracle. SG's attractiveness is due in part to its simplicity of executing a single step along the negative subsampled gradient direction to update the incumbent iterate. In this paper, we question SG's choice of executing a single step as opposed to multiple steps between subsample updates. Our investigation leads naturally to generalizing SG into Retrospective Approximation (RA) where, during each iteration, a "deterministic solver" executes possibly multiple steps on a subsampled deterministic problem and stops when further solving is deemed unnecessary from the standpoint of statistical efficiency. RA thus rigorizes what is appealing for implementation -- during each iteration, "plug in" a solver, e.g., L-BFGS line search or Newton-CG, as is, and solve only to the extent necessary. We develop a complete theory using relative error of the observed gradients as the principal object, demonstrating that almost sure and $L_1$ consistency of RA are preserved under especially weak conditions when sample sizes are increased at appropriate rates. We also characterize the iteration and oracle complexity (for linear and sub-linear solvers) of RA, and identify a practical termination criterion leading to optimal complexity rates. To subsume non-convex $f$, we present a certain "random central limit theorem" that incorporates the effect of curvature across all first-order critical points, demonstrating that the asymptotic behavior is described by a certain mixture of normals. The message from our numerical experiments is that the ability of RA to incorporate existing second-order deterministic solvers in a strategic manner might be important from the standpoint of dispensing with hyper-parameter tuning.

Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

Stochastic Sub-Sampled Newton Method with Variance Reduction

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

A stochastic variance reduced gradient method with adaptive step for stochastic optimization

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Bound Analysis of Natural Gradient Descent in Stochastic Optimization Setting

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Convergence of Riemannian Stochastic Gradient Descent on Hadamard Manifold

Variance-Reduced Proximal Stochastic Gradient Descent for Non-convex Composite optimization.

An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization

New Insight into Hybrid Stochastic Gradient Descent: Beyond With-Replacement Sampling and Convexity.

Painless Stochastic Conjugate Gradient for Large-Scale Machine Learning

VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning

Larger is Better: The Effect of Learning Rates Enjoyed by Stochastic Optimization with Progressive Variance Reduction

A Retrospective Approximation Approach for Smooth Stochastic Optimization

Parallel Asynchronous Stochastic Variance Reduction for Nonconvex Optimization

Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems