Abstract:Adaptive gradient methods, such as AdaGrad, are among the most successful optimization algorithms for neural network training. While these methods are known to achieve better dimensional dependence than stochastic gradient descent (SGD) under favorable geometry for stochastic convex optimization, the theoretical justification for their success in stochastic non-convex optimization remains elusive. In fact, under standard assumptions of Lipschitz gradients and bounded noise variance, it is known that SGD is worst-case optimal (up to absolute constants) in terms of finding a near-stationary point with respect to the $\ell_2$-norm, making further improvements impossible. Motivated by this limitation, we introduce refined assumptions on the smoothness structure of the objective and the gradient noise variance, which better suit the coordinate-wise nature of adaptive gradient methods. Moreover, we adopt the $\ell_1$-norm of the gradient as the stationarity measure, as opposed to the standard $\ell_2$-norm, to align with the coordinate-wise analysis and obtain tighter convergence guarantees for AdaGrad. Under these new assumptions and the $\ell_1$-norm stationarity measure, we establish an upper bound on the convergence rate of AdaGrad and a corresponding lower bound for SGD. In particular, for certain configurations of problem parameters, we show that the iteration complexity of AdaGrad outperforms SGD by a factor of $d$. To the best of our knowledge, this is the first result to demonstrate a provable gain of adaptive gradient methods over SGD in a non-convex setting. We also present supporting lower bounds, including one specific to AdaGrad and one applicable to general deterministic first-order methods, showing that our upper bound for AdaGrad is tight and unimprovable up to a logarithmic factor under certain conditions.

A Proof of Exact Convergence Rate of Gradient Descent. Part I. Performance Criterion $\Vert \nabla f(x_N)\Vert^2/(f(x_0)-f_*)$

A Proof of Exact Convergence Rate of Gradient Descent. Part II. Performance Criterion $(f(x_N)-f_)/\|x_0-x_\|^2$

Exact worst-case convergence rates of gradient descent: a complete analysis for all constant stepsizes over nonconvex and convex functions

On the worst-case complexity of the gradient method with exact line search for smooth strongly convex functions

Inexact Riemannian Gradient Descent Method for Nonconvex Optimization

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

Exact convergence rate of the last iterate in subgradient methods

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

The Average Rate of Convergence of the Exact Line Search Gradient Descent Method

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Provably Faster Gradient Descent via Long Steps

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

Derivatives of Stochastic Gradient Descent in parametric optimization

A new convergence rate of the steepest descent regarding the Euclidean norm

Exact Linear Convergence Rate Analysis for Low-Rank Symmetric Matrix Completion via Gradient Descent

Intermediate Gradient Methods with Relative Inexactness