Abstract: It is not clear yet why ADAM-alike adaptive gradient algorithms suffer from worse generalization performance than SGD despite their faster training speed. This work aims to provide understandings on this generalization gap by analyzing their local convergence behaviors. Specifically, we observe the heavy tails of gradient noise in these algorithms. This motivates us to analyze these algorithms through their Levy-driven stochastic differential equations (SDEs) because of the similar convergence behaviors of an algorithm and its SDE. Then we establish the escaping time of these SDEs from a local basin. The result shows that (1) the escaping time of both SGD and ADAM~depends on the Radon measure of the basin positively and the heaviness of gradient noise negatively; (2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainly because (a) the geometry adaptation in ADAM~via adaptively scaling each gradient coordinate well diminishes the anisotropic structure in gradient noise and results in larger Radon measure of a basin; (b) the exponential gradient average in ADAM~smooths its gradient and leads to lighter gradient noise tails than SGD. So SGD is more locally unstable than ADAM~at sharp minima defined as the minima whose local basins have small Radon measure, and can better escape from them to flatter ones with larger Radon measure. As flat minima here which often refer to the minima at flat or asymmetric basins/valleys often generalize better than sharp ones~\cite{keskar2016large,he2019asymmetric}, our result explains the better generalization performance of SGD over ADAM. Finally, experimental results confirm our heavy-tailed gradient noise assumption and theoretical affirmation.

Defying the Gravity of Learning Curve: a Characteristic of Nearest Neighbour Anomaly Detectors

Algorithms that Defy the Gravity of Learning Curve

Adaptive-Gravity: A Defense Against Adversarial Samples

Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks

A Real Negative Selection Algorithm with Evolutionary Preference for Anomaly Detection

Anomaly Detection via Neighbourhood Contrast.

Gravitational Fixed Radius Nearest Neighbor for Imbalanced Problem

Learning to Detect Interesting Anomalies

Learning Minimum Volume Sets and Anomaly Detectors from KNN Graphs

Anomaly Detection using Principles of Human Perception

NNS:A novel neighborhood negative selection algorithm

k-NNN: Nearest Neighbors of Neighbors for Anomaly Detection

On The Relationship between Visual Anomaly-free and Anomalous Representations

Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning

Cloud-based machine learning algorithms for anomalies detection

Two Phases of Scaling Laws for Nearest Neighbor Classifiers

Learning Competitive and Discriminative Reconstructions for Anomaly Detection

Entropy and Gravitation Based Dynamic Radius Nearest Neighbor Classification for Imbalanced Problem

Multiple Descents in Unsupervised Learning: The Role of Noise, Domain Shift and Anomalies

On the gravitation‐based classification: A novel algorithm using equilibrium points for enhanced learning and dimensionality reduction

Anomaly Detection Using Neighborhood Negative Selection