Abstract:Adaptive gradient methods, such as AdaGrad, are among the most successful optimization algorithms for neural network training. While these methods are known to achieve better dimensional dependence than stochastic gradient descent (SGD) under favorable geometry for stochastic convex optimization, the theoretical justification for their success in stochastic non-convex optimization remains elusive. In fact, under standard assumptions of Lipschitz gradients and bounded noise variance, it is known that SGD is worst-case optimal (up to absolute constants) in terms of finding a near-stationary point with respect to the $\ell_2$-norm, making further improvements impossible. Motivated by this limitation, we introduce refined assumptions on the smoothness structure of the objective and the gradient noise variance, which better suit the coordinate-wise nature of adaptive gradient methods. Moreover, we adopt the $\ell_1$-norm of the gradient as the stationarity measure, as opposed to the standard $\ell_2$-norm, to align with the coordinate-wise analysis and obtain tighter convergence guarantees for AdaGrad. Under these new assumptions and the $\ell_1$-norm stationarity measure, we establish an upper bound on the convergence rate of AdaGrad and a corresponding lower bound for SGD. In particular, for certain configurations of problem parameters, we show that the iteration complexity of AdaGrad outperforms SGD by a factor of $d$. To the best of our knowledge, this is the first result to demonstrate a provable gain of adaptive gradient methods over SGD in a non-convex setting. We also present supporting lower bounds, including one specific to AdaGrad and one applicable to general deterministic first-order methods, showing that our upper bound for AdaGrad is tight and unimprovable up to a logarithmic factor under certain conditions.

Gradient Descent in the Absence of Global Lipschitz Continuity of the Gradients

On the Unstable Convergence Regime of Gradient Descent

Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness

Super Gradient Descent: Global Optimization requires Global Gradient

Directional Smoothness and Gradient Methods: Convergence and Adaptivity

Beyond the Edge of Stability via Two-step Gradient Updates

Toward a Unified Theory of Gradient Descent under Generalized Smoothness

Concavifiability and convergence: necessary and sufficient conditions for gradient descent analysis

A gradient descent akin method for constrained optimization: algorithms and applications

Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes

On Linear Convergence of Non-Euclidean Gradient Methods Without Strong Convexity and Lipschitz Gradient Continuity.

Non-Uniform Smoothness for Gradient Descent

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency

Gradient is All You Need?

Generalization to the Natural Gradient Descent

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

First Analysis of Local GD on Heterogeneous Data

Stochastic Gradient Descent in the Viewpoint of Graduated Optimization

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps