Abstract:Stochastic Gradient (SG) is the defacto iterative technique to solve stochastic optimization (SO) problems with a smooth (non-convex) objective $f$ and a stochastic first-order oracle. SG's attractiveness is due in part to its simplicity of executing a single step along the negative subsampled gradient direction to update the incumbent iterate. In this paper, we question SG's choice of executing a single step as opposed to multiple steps between subsample updates. Our investigation leads naturally to generalizing SG into Retrospective Approximation (RA) where, during each iteration, a "deterministic solver" executes possibly multiple steps on a subsampled deterministic problem and stops when further solving is deemed unnecessary from the standpoint of statistical efficiency. RA thus rigorizes what is appealing for implementation -- during each iteration, "plug in" a solver, e.g., L-BFGS line search or Newton-CG, as is, and solve only to the extent necessary. We develop a complete theory using relative error of the observed gradients as the principal object, demonstrating that almost sure and $L_1$ consistency of RA are preserved under especially weak conditions when sample sizes are increased at appropriate rates. We also characterize the iteration and oracle complexity (for linear and sub-linear solvers) of RA, and identify a practical termination criterion leading to optimal complexity rates. To subsume non-convex $f$, we present a certain "random central limit theorem" that incorporates the effect of curvature across all first-order critical points, demonstrating that the asymptotic behavior is described by a certain mixture of normals. The message from our numerical experiments is that the ability of RA to incorporate existing second-order deterministic solvers in a strategic manner might be important from the standpoint of dispensing with hyper-parameter tuning.

A Stochastic Iteratively Regularized Gauss-Newton Method

On a Dynamic Variant of the Iteratively Regularized Gauss-Newton Method with Sequential Data.

A Graph-based Iterative Algorithm for Inverse Problems with Local and Non-local Regularizations

A Proximal Iteratively Regularized Gauss–Newton Method for Nonlinear Inverse Problems

Stochastic Sub-Sampled Newton Method with Variance Reduction

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

A convergence analysis of the iteratively regularized Gauss-Newton method under Lipschitz condition

Stochastic Data-Driven Bouligand Landweber Method for Solving Non-smooth Inverse Problems

On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

A stochastic regularized second-order iterative scheme for optimal control and inverse problems in stochastic partial differential equations

Analysis of a Generalized Regularized Gauss–Newton Method under Heuristic Rule in Banach Spaces

Stochastic gradient descent for linear inverse problems in variable exponent Lebesgue spaces

Hanke-Raus heuristic rule for iteratively regularized stochastic gradient descent

Stochastic Gradient Descent Method with Convex Penalty for Ill-Posed Problems in Banach Spaces

Convergence and adaptive discretization of the IRGNM Tikhonov and the IRGNM Ivanov method under a tangential cone condition in Banach space

A Retrospective Approximation Approach for Smooth Stochastic Optimization

Super-Universal Regularized Newton Method

RES: Regularized Stochastic BFGS Algorithm

Application of the iteratively regularized Gauss–Newton method to parameter identification problems in Computational Fluid Dynamics

Stochastic Gauss–Seidel type inertial proximal alternating linearized minimization and its application to proximal neural networks

Randomized Matrix Approximation to Enhance Regularized Projection Schemes in Inverse Problems