Abstract:Stochastic Gradient (SG) is the defacto iterative technique to solve stochastic optimization (SO) problems with a smooth (non-convex) objective $f$ and a stochastic first-order oracle. SG's attractiveness is due in part to its simplicity of executing a single step along the negative subsampled gradient direction to update the incumbent iterate. In this paper, we question SG's choice of executing a single step as opposed to multiple steps between subsample updates. Our investigation leads naturally to generalizing SG into Retrospective Approximation (RA) where, during each iteration, a "deterministic solver" executes possibly multiple steps on a subsampled deterministic problem and stops when further solving is deemed unnecessary from the standpoint of statistical efficiency. RA thus rigorizes what is appealing for implementation -- during each iteration, "plug in" a solver, e.g., L-BFGS line search or Newton-CG, as is, and solve only to the extent necessary. We develop a complete theory using relative error of the observed gradients as the principal object, demonstrating that almost sure and $L_1$ consistency of RA are preserved under especially weak conditions when sample sizes are increased at appropriate rates. We also characterize the iteration and oracle complexity (for linear and sub-linear solvers) of RA, and identify a practical termination criterion leading to optimal complexity rates. To subsume non-convex $f$, we present a certain "random central limit theorem" that incorporates the effect of curvature across all first-order critical points, demonstrating that the asymptotic behavior is described by a certain mixture of normals. The message from our numerical experiments is that the ability of RA to incorporate existing second-order deterministic solvers in a strategic manner might be important from the standpoint of dispensing with hyper-parameter tuning.

Zeroth-Order Riemannian Averaging Stochastic Approximation Algorithms

Zeroth-order Riemannian Averaging Stochastic Approximation Algorithms

Riemannian Accelerated Zeroth-order Algorithm: Improved Robustness and Lower Query Complexity

Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization

Zeroth-order Random Subspace Algorithm for Non-smooth Convex Optimization

Stochastic Zeroth-order Optimization Via Variance Reduction Method.

Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming

Zeroth-order algorithms for stochastic distributed nonconvex optimization

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

Robust Hybrid Zero-Order Optimization Algorithms with Acceleration via Averaging in Time

A Dimension-Insensitive Algorithm for Stochastic Zeroth-Order Optimization

Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds

Obtaining Lower Query Complexities Through Lightweight Zeroth-Order Proximal Gradient Algorithms

A Retrospective Approximation Approach for Smooth Stochastic Optimization

Riemannian stochastic fixed point optimization algorithm

A Zeroth-Order Variance-Reduced Method for Decentralized Stochastic Non-convex Optimization

Small Errors in Random Zeroth-Order Optimization Are Imaginary

Zero-Order Stochastic Conditional Gradient Sliding Method for Non-smooth Convex Optimization