Abstract:Adaptive gradient methods, such as AdaGrad, are among the most successful optimization algorithms for neural network training. While these methods are known to achieve better dimensional dependence than stochastic gradient descent (SGD) under favorable geometry for stochastic convex optimization, the theoretical justification for their success in stochastic non-convex optimization remains elusive. In fact, under standard assumptions of Lipschitz gradients and bounded noise variance, it is known that SGD is worst-case optimal (up to absolute constants) in terms of finding a near-stationary point with respect to the $\ell_2$-norm, making further improvements impossible. Motivated by this limitation, we introduce refined assumptions on the smoothness structure of the objective and the gradient noise variance, which better suit the coordinate-wise nature of adaptive gradient methods. Moreover, we adopt the $\ell_1$-norm of the gradient as the stationarity measure, as opposed to the standard $\ell_2$-norm, to align with the coordinate-wise analysis and obtain tighter convergence guarantees for AdaGrad. Under these new assumptions and the $\ell_1$-norm stationarity measure, we establish an upper bound on the convergence rate of AdaGrad and a corresponding lower bound for SGD. In particular, for certain configurations of problem parameters, we show that the iteration complexity of AdaGrad outperforms SGD by a factor of $d$. To the best of our knowledge, this is the first result to demonstrate a provable gain of adaptive gradient methods over SGD in a non-convex setting. We also present supporting lower bounds, including one specific to AdaGrad and one applicable to general deterministic first-order methods, showing that our upper bound for AdaGrad is tight and unimprovable up to a logarithmic factor under certain conditions.

Hölder Gradient Descent and Adaptive Regularization Methods in Banach Spaces for First-Order Points

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

High-Probability Bound for Non-Smooth Non-Convex Stochastic Optimization with Heavy Tails

Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods

A Stochastic Objective-Function-Free Adaptive Regularization Method with Optimal Complexity

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

The Randomized Block Coordinate Descent Method in the Hölder Smooth Setting

Non-Uniform Smoothness for Gradient Descent

Faster Acceleration for Steepest Descent

Asymptotic error analysis for stochastic gradient optimization schemes with first and second order modified equations

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces

Dealing with unbounded gradients in stochastic saddle-point optimization

Non-Euclidean High-Order Smooth Convex Optimization

Inexact Newton Regularization in Banach Spaces Based on Two-Point Gradient Method with Uniformly Convex Penalty Terms

Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization

First-Order Methods for Nonsmooth Nonconvex Functional Constrained Optimization with or without Slater Points

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization