Abstract:We present adaptive gradient methods (both basic and accelerated) for solving convex composite optimization problems in which the main part is approximately smooth (a.k.a. $(\delta, L)$-smooth) and can be accessed only via a (potentially biased) stochastic gradient oracle. This setting covers many interesting examples including Hölder smooth problems and various inexact computations of the stochastic gradient. Our methods use AdaGrad stepsizes and are adaptive in the sense that they do not require knowing any problem-dependent constants except an estimate of the diameter of the feasible set but nevertheless achieve the best possible convergence rates as if they knew the corresponding constants. We demonstrate that AdaGrad stepsizes work in a variety of situations by proving, in a unified manner, three types of new results. First, we establish efficiency guarantees for our methods in the classical setting where the oracle's variance is uniformly bounded. We then show that, under more refined assumptions on the variance, the same methods without any modifications enjoy implicit variance reduction properties allowing us to express their complexity estimates in terms of the variance only at the minimizer. Finally, we show how to incorporate explicit SVRG-type variance reduction into our methods and obtain even faster algorithms. In all three cases, we present both basic and accelerated algorithms achieving state-of-the-art complexity bounds. As a direct corollary of our results, we obtain universal stochastic gradient methods for Hölder smooth problems which can be used in all situations.

A new inexact stochastic recursive gradient descent algorithm with Barzilai–Borwein step size in machine learning

A Modified Inexact SARAH Algorithm with Stabilized Barzilai-Borwein Step-Size in Machine learning

Accelerating Mini-batch SARAH by Step Size Rules

Barzilai-Borwein Step Size for Stochastic Gradient Descent

Adaptive step size rules for stochastic optimization in large-scale learning

A variable metric mini-batch proximal stochastic recursive gradient algorithm with diagonal Barzilai-Borwein stepsize

A stochastic variance reduced gradient method with adaptive step for stochastic optimization

Random-reshuffled SARAH does not need a full gradient computations

AdaBB: Adaptive Barzilai-Borwein Method for Convex Optimization

A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information

Variable Metric Proximal Gradient Method with Diagonal Barzilai-Borwein Stepsize

Sublinear Regret with Barzilai-Borwein Step Sizes

Gradient Methods with Adaptive Step-Sizes

Variational Barzilai-Borwein Method

A stochastic two-step inertial Bregman proximal alternating linearized minimization algorithm for nonconvex and nonsmooth problems

An Adaptive Incremental Gradient Method With Support for Non-Euclidean Norms

History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms

SAAGs: Biased stochastic variance reduction methods for large-scale learning

On the acceleration of the Barzilai-Borwein method

Regularized Barzilai-Borwein method

Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction