Abstract:Stochastic Gradient (SG) is the defacto iterative technique to solve stochastic optimization (SO) problems with a smooth (non-convex) objective $f$ and a stochastic first-order oracle. SG's attractiveness is due in part to its simplicity of executing a single step along the negative subsampled gradient direction to update the incumbent iterate. In this paper, we question SG's choice of executing a single step as opposed to multiple steps between subsample updates. Our investigation leads naturally to generalizing SG into Retrospective Approximation (RA) where, during each iteration, a "deterministic solver" executes possibly multiple steps on a subsampled deterministic problem and stops when further solving is deemed unnecessary from the standpoint of statistical efficiency. RA thus rigorizes what is appealing for implementation -- during each iteration, "plug in" a solver, e.g., L-BFGS line search or Newton-CG, as is, and solve only to the extent necessary. We develop a complete theory using relative error of the observed gradients as the principal object, demonstrating that almost sure and $L_1$ consistency of RA are preserved under especially weak conditions when sample sizes are increased at appropriate rates. We also characterize the iteration and oracle complexity (for linear and sub-linear solvers) of RA, and identify a practical termination criterion leading to optimal complexity rates. To subsume non-convex $f$, we present a certain "random central limit theorem" that incorporates the effect of curvature across all first-order critical points, demonstrating that the asymptotic behavior is described by a certain mixture of normals. The message from our numerical experiments is that the ability of RA to incorporate existing second-order deterministic solvers in a strategic manner might be important from the standpoint of dispensing with hyper-parameter tuning.

A Note on the Stochastic Ruler Method for Discrete Simulation Optimization

Stochastic Steepest-Descent Optimization Of Multiple-Objective Mobile Sensor Coverage

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Simulation-Based Discrete Optimization of Stochastic Discrete Event Systems Subject to Non Closed-Form Constraints.

A Unified Analysis of Stochastic Optimization Methods Using Jump System Theory and Quadratic Constraints

On the rates of convergence of simulation based optimization algorithms for optimal stopping problems

A Stochastic Record-Value Approach to Global Simulation Optimization

Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization

A Retrospective Approximation Approach for Smooth Stochastic Optimization

Discrete optimization via simulation: algorithms and error control

Asymptotic Optimality in Stochastic Optimization

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

Constrained Stochastic Recursive Momentum Successive Convex Approximation

Stochastic Optimization with Constraints: A Non-asymptotic Instance-Dependent Analysis

Balancing Search and Estimation in Random Search Based Stochastic Simulation Optimization

A Stochastic Approximation Method for Simulation-Based Quantile Optimization

Optimization Via Adaptive Sampling and Regenerative Simulation.

A Multilevel Approach for Stochastic Nonlinear Optimal Control