Abstract:Adaptive gradient methods, such as AdaGrad, are among the most successful optimization algorithms for neural network training. While these methods are known to achieve better dimensional dependence than stochastic gradient descent (SGD) under favorable geometry for stochastic convex optimization, the theoretical justification for their success in stochastic non-convex optimization remains elusive. In fact, under standard assumptions of Lipschitz gradients and bounded noise variance, it is known that SGD is worst-case optimal (up to absolute constants) in terms of finding a near-stationary point with respect to the $\ell_2$-norm, making further improvements impossible. Motivated by this limitation, we introduce refined assumptions on the smoothness structure of the objective and the gradient noise variance, which better suit the coordinate-wise nature of adaptive gradient methods. Moreover, we adopt the $\ell_1$-norm of the gradient as the stationarity measure, as opposed to the standard $\ell_2$-norm, to align with the coordinate-wise analysis and obtain tighter convergence guarantees for AdaGrad. Under these new assumptions and the $\ell_1$-norm stationarity measure, we establish an upper bound on the convergence rate of AdaGrad and a corresponding lower bound for SGD. In particular, for certain configurations of problem parameters, we show that the iteration complexity of AdaGrad outperforms SGD by a factor of $d$. To the best of our knowledge, this is the first result to demonstrate a provable gain of adaptive gradient methods over SGD in a non-convex setting. We also present supporting lower bounds, including one specific to AdaGrad and one applicable to general deterministic first-order methods, showing that our upper bound for AdaGrad is tight and unimprovable up to a logarithmic factor under certain conditions.

Adversarial Training for Gradient Descent: Analysis Through its Continuous-time Approximation

Gradient Adversarial Training of Neural Networks

Reducing Adversarial Training Cost with Gradient Approximation

Analysis of Stochastic Gradient Descent in Continuous Time

On Gradient-Based Learning in Continuous Games

Continuous Time Analysis of Momentum Methods

Two-Timescale Gradient Descent Ascent Algorithms for Nonconvex Minimax Optimization

You Only Propagate Once: Accelerating Adversarial Training Via Maximal Principle

You Only Propagate Once: Painless Adversarial Training Using Maximal Principle

Stability Analysis and Generalization Bounds of Adversarial Training

Stochastic Gradient Descent in Continuous Time: A Central Limit Theorem

Non-asymptotic convergence analysis of the stochastic gradient Hamiltonian Monte Carlo algorithm with discontinuous stochastic gradient with applications to training of ReLU neural networks

The Anytime Convergence of Stochastic Gradient Descent with Momentum: From a Continuous-Time Perspective

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

Approximation and Gradient Descent Training with Neural Networks

Quantifying the Preferential Direction of the Model Gradient in Adversarial Training With Projected Gradient Descent

L G ] 1 9 Ju n 20 19 Convergence of Adversarial Training in Overparametrized Networks

Continuous-Time Analysis of Adaptive Optimization and Normalization

Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation

Convergence Rates of Accelerated Markov Gradient Descent with Applications in Reinforcement Learning

Adversarial Training Methods for Deep Learning: A Systematic Review