Abstract: It is not clear yet why ADAM-alike adaptive gradient algorithms suffer from worse generalization performance than SGD despite their faster training speed. This work aims to provide understandings on this generalization gap by analyzing their local convergence behaviors. Specifically, we observe the heavy tails of gradient noise in these algorithms. This motivates us to analyze these algorithms through their Levy-driven stochastic differential equations (SDEs) because of the similar convergence behaviors of an algorithm and its SDE. Then we establish the escaping time of these SDEs from a local basin. The result shows that (1) the escaping time of both SGD and ADAM~depends on the Radon measure of the basin positively and the heaviness of gradient noise negatively; (2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainly because (a) the geometry adaptation in ADAM~via adaptively scaling each gradient coordinate well diminishes the anisotropic structure in gradient noise and results in larger Radon measure of a basin; (b) the exponential gradient average in ADAM~smooths its gradient and leads to lighter gradient noise tails than SGD. So SGD is more locally unstable than ADAM~at sharp minima defined as the minima whose local basins have small Radon measure, and can better escape from them to flatter ones with larger Radon measure. As flat minima here which often refer to the minima at flat or asymmetric basins/valleys often generalize better than sharp ones~\cite{keskar2016large,he2019asymmetric}, our result explains the better generalization performance of SGD over ADAM. Finally, experimental results confirm our heavy-tailed gradient noise assumption and theoretical affirmation.

Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning

The Regularization Effects of Anisotropic Noise in Stochastic Gradient Descent.

Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients

Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be

Improving Generalization Performance of Adaptive Gradient Method Via Bounded Step Sizes

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD

Why (and When) does Local SGD Generalize Better than SGD?

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Stochastic Gradient Descent Introduces an Effective Landscape-Dependent Regularization Favoring Flat Solutions

Why ADAM Beats SGD for Attention Models

Stability-Based Generalization Analysis of the Asynchronous Decentralized SGD

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects

Understanding Short-Range Memory Effects in Deep Neural Networks

Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization

Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent

Normalized Direction-preserving Adam.

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks

AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations

AdaGrad under Anisotropic Smoothness

The Implicit Regularization for Adaptive Optimization Algorithms on Homogeneous Neural Networks