Abstract:Adaptive gradient methods, such as AdaGrad, are among the most successful optimization algorithms for neural network training. While these methods are known to achieve better dimensional dependence than stochastic gradient descent (SGD) under favorable geometry for stochastic convex optimization, the theoretical justification for their success in stochastic non-convex optimization remains elusive. In fact, under standard assumptions of Lipschitz gradients and bounded noise variance, it is known that SGD is worst-case optimal (up to absolute constants) in terms of finding a near-stationary point with respect to the $\ell_2$-norm, making further improvements impossible. Motivated by this limitation, we introduce refined assumptions on the smoothness structure of the objective and the gradient noise variance, which better suit the coordinate-wise nature of adaptive gradient methods. Moreover, we adopt the $\ell_1$-norm of the gradient as the stationarity measure, as opposed to the standard $\ell_2$-norm, to align with the coordinate-wise analysis and obtain tighter convergence guarantees for AdaGrad. Under these new assumptions and the $\ell_1$-norm stationarity measure, we establish an upper bound on the convergence rate of AdaGrad and a corresponding lower bound for SGD. In particular, for certain configurations of problem parameters, we show that the iteration complexity of AdaGrad outperforms SGD by a factor of $d$. To the best of our knowledge, this is the first result to demonstrate a provable gain of adaptive gradient methods over SGD in a non-convex setting. We also present supporting lower bounds, including one specific to AdaGrad and one applicable to general deterministic first-order methods, showing that our upper bound for AdaGrad is tight and unimprovable up to a logarithmic factor under certain conditions.

Convergence results for gradient flow and gradient descent systems in the artificial neural network training

A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Novel Convergence Results of Adaptive Stochastic Gradient Descents

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

Normalized gradient flow optimization in the training of ReLU artificial neural networks

Convergence of continuous-time stochastic gradient descent with applications to linear deep neural networks

A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions

Convergence of Stochastic Gradient Descent in Deep Neural Network

Convergence of Gradient Method for A Fully Recurrent Neural Network

A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks

On the Convergence of Gradient Descent for Large Learning Rates

Convergence of batch gradient algorithm for feedforward neural network training

Convergence Analysis of a Class of Nonsmooth Gradient Systems.

Convergence Dynamics of Generative Adversarial Networks: The Dual Metric Flows

On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks

Approximation and Gradient Descent Training with Neural Networks

Convergence of Batch Gradient Learning with Smoothing Regularization and Adaptive Momentum for Neural Networks

On Convergence of Training Loss Without Reaching Stationary Points

On the Convergence of Decentralized Adaptive Gradient Methods