Abstract:We here introduce Multi-Iteration Stochastic Optimizers, a novel class of first-order stochastic optimizers where the relative $L^2$ error is estimated and controlled using successive control variates along the path of iterations. By exploiting the correlation between iterates, control variates may reduce the estimator's variance so that an accurate estimation of the mean gradient becomes computationally affordable. We name the estimator of the mean gradient Multi-Iteration stochastiC Estimator (MICE). In principle, MICE can be flexibly coupled with any first-order stochastic optimizer, given its non-intrusive nature. Our generic algorithm adaptively decides which iterates to keep in its index set. We present an error analysis of MICE and a convergence analysis of Multi-Iteration Stochastic Optimizers for different classes of problems, including some non-convex cases. Within the smooth, strongly convex setting, we show that to approximate a minimizer with accuracy $tol$, SGD-MICE requires, on average, $O(tol^{-1})$ stochastic gradient evaluations, while SGD with adaptive batch sizes requires $O(tol^{-1} \log(tol^{-1}))$, correspondingly. Moreover, in a numerical evaluation, SGD-MICE achieved tol with less than 3% the number of gradient evaluations than adaptive batch SGD. The MICE estimator provides a straightforward stopping criterion based on the gradient norm that is validated in consistency tests. To assess the efficiency of MICE, we present several examples in which we use SGD-MICE and Adam-MICE. We include one example based on a stochastic adaptation of the Rosenbrock function and logistic regression training for various datasets. When compared to SGD, SAG, SAGA, SVRG, and SARAH, the Multi-Iteration Stochastic Optimizers reduced, without the need to tune parameters for each example, the gradient sampling cost in all cases tested, also being competitive in runtime in some cases.

Multi-index antithetic stochastic gradient algorithm

Accelerated stochastic approximation with state-dependent noise

A Mini-Batch Stochastic Conjugate Gradient Algorithm with Variance Reduction.

Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation

Stochastic Average Gradient : A Simple Empirical Investigation

Stochastic Gradient Markov Chain Monte Carlo

A Stochastic GDA Method With Backtracking For Solving Nonconvex (Strongly) Concave Minimax Problems

SGD-rα: A Real-Time Α-Suffix Averaging Method for SGD with Biased Gradient Estimates

On Biased Stochastic Gradient Estimation

Multi-Iteration Stochastic Optimizers

Stochastic Approximate Gradient Descent via the Langevin Algorithm

Painless Stochastic Conjugate Gradient for Large-Scale Machine Learning

Robust Approximate Sampling via Stochastic Gradient Barker Dynamics

A Hybrid Stochastic Gradient Hamiltonian Monte Carlo Method

Convergence Analysis of Stochastic Gradient Descent with MCMC Estimators

Nonasymptotic analysis of Stochastic Gradient Hamiltonian Monte Carlo under local conditions for nonconvex optimization

$μ^2$-SGD: Stable Stochastic Optimization via a Double Momentum Mechanism

A Retrospective Approximation Approach for Smooth Stochastic Optimization

An adaptive Hessian approximated stochastic gradient MCMC method

A Simple Stochastic Variance Reduced Algorithm with Fast Convergence Rates.

On the Bias-Variance Tradeoff in Stochastic Gradient Methods