Abstract:Adaptive gradient methods, such as AdaGrad, are among the most successful optimization algorithms for neural network training. While these methods are known to achieve better dimensional dependence than stochastic gradient descent (SGD) under favorable geometry for stochastic convex optimization, the theoretical justification for their success in stochastic non-convex optimization remains elusive. In fact, under standard assumptions of Lipschitz gradients and bounded noise variance, it is known that SGD is worst-case optimal (up to absolute constants) in terms of finding a near-stationary point with respect to the $\ell_2$-norm, making further improvements impossible. Motivated by this limitation, we introduce refined assumptions on the smoothness structure of the objective and the gradient noise variance, which better suit the coordinate-wise nature of adaptive gradient methods. Moreover, we adopt the $\ell_1$-norm of the gradient as the stationarity measure, as opposed to the standard $\ell_2$-norm, to align with the coordinate-wise analysis and obtain tighter convergence guarantees for AdaGrad. Under these new assumptions and the $\ell_1$-norm stationarity measure, we establish an upper bound on the convergence rate of AdaGrad and a corresponding lower bound for SGD. In particular, for certain configurations of problem parameters, we show that the iteration complexity of AdaGrad outperforms SGD by a factor of $d$. To the best of our knowledge, this is the first result to demonstrate a provable gain of adaptive gradient methods over SGD in a non-convex setting. We also present supporting lower bounds, including one specific to AdaGrad and one applicable to general deterministic first-order methods, showing that our upper bound for AdaGrad is tight and unimprovable up to a logarithmic factor under certain conditions.

Convergence Analysis of Three Classes of Split-Complex Gradient Algorithms for Complex-Valued Recurrent Neural Networks

Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks

Convergence Analysis of Batch Gradient Algorithm for Three Classes of Sigma-Pi Neural Networks

Deterministic Convergence of Complex Mini-Batch Gradient Learning Algorithm for Fully Complex-Valued Neural Networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Convergence Analysis of the Learning Algorithms for Multi-Valued Neurons

Convergence Analysis of Graph Regularized Non-Negative Matrix Factorization

Convergence Analysis of Caputo-Type Fractional Order Complex-Valued Neural Networks

A Hybrid Complex Spectral Conjugate Gradient Learning Algorithm for Complex-Valued Data Processing

Convergence Analysis of Gradient Neural Network Solving Linear Matrix Equations

Convergence Analysis of a Class of Nonsmooth Gradient Systems.

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Convergence analysis of an augmented algorithm for fully complex-valued neural networks

A Split-Complex Valued Gradient-Based Descent Neuro-Fuzzy Algorithm for TS System and Its Convergence

Convergence of Gradient Method for A Fully Recurrent Neural Network

Deterministic convergence of chaos injection-based gradient method for training feedforward neural networks.

Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions

A complex conjugate gradient training algorithm with Barzilai-Borwein stepsize for complex-valued neural networks

Fully Complex Conjugate Gradient-Based Neural Networks Using Wirtinger Calculus Framework: Deterministic Convergence and Its Application.

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions