Abstract:Adaptive gradient methods, such as AdaGrad, are among the most successful optimization algorithms for neural network training. While these methods are known to achieve better dimensional dependence than stochastic gradient descent (SGD) under favorable geometry for stochastic convex optimization, the theoretical justification for their success in stochastic non-convex optimization remains elusive. In fact, under standard assumptions of Lipschitz gradients and bounded noise variance, it is known that SGD is worst-case optimal (up to absolute constants) in terms of finding a near-stationary point with respect to the $\ell_2$-norm, making further improvements impossible. Motivated by this limitation, we introduce refined assumptions on the smoothness structure of the objective and the gradient noise variance, which better suit the coordinate-wise nature of adaptive gradient methods. Moreover, we adopt the $\ell_1$-norm of the gradient as the stationarity measure, as opposed to the standard $\ell_2$-norm, to align with the coordinate-wise analysis and obtain tighter convergence guarantees for AdaGrad. Under these new assumptions and the $\ell_1$-norm stationarity measure, we establish an upper bound on the convergence rate of AdaGrad and a corresponding lower bound for SGD. In particular, for certain configurations of problem parameters, we show that the iteration complexity of AdaGrad outperforms SGD by a factor of $d$. To the best of our knowledge, this is the first result to demonstrate a provable gain of adaptive gradient methods over SGD in a non-convex setting. We also present supporting lower bounds, including one specific to AdaGrad and one applicable to general deterministic first-order methods, showing that our upper bound for AdaGrad is tight and unimprovable up to a logarithmic factor under certain conditions.

Convergence Analysis of a Class of Nonsmooth Gradient Systems.

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Dynamical behavior of a class of nonsmooth gradient-like systems

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Convergence and non-convergence in a nonlocal gradient flow

Dynamical Analysis of Neural Networks of Subgradient System

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

A convergence result of a continuous model of deep learning via Łojasiewicz--Simon inequality

Subgradient-based Neural Networks for Nonsmooth Nonconvex Optimization Problems

Convergence of Newton'S Method for Systems of Equations with Constant Rank Derivatives

Convergence Criterion of Newton's Method for Singular Systems with Constant Rank Derivatives

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Generalized Gradient Projection Neural Networks for Nonsmooth Optimization Problems

Convex and Non-convex Optimization Under Generalized Smoothness

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Convergence of Gradient Algorithms for Nonconvex C1+α Cost Functions

Convergence of analytic gradient-type systems with periodicity and its applications in Kuramoto models.

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

Convergence Rate Analysis of Continuous- and Discrete-Time Smoothing Gradient Algorithms

The Long Time Behavior of A Class of Second-Order Gradient-Like Systems with Vanishing Dissipative Term and Non-Convex Analytic Potential

Convergence Properties of Nonlinear Conjugate Gradient Methods.