Abstract:Deep learning algorithms - typically consisting of a class of deep neural networks trained by a stochastic gradient descent (SGD) optimization method - are nowadays the key ingredients in many artificial intelligence (AI) systems and have revolutionized our ways of working and living in modern societies. For example, SGD methods are used to train powerful large language models (LLMs) such as versions of ChatGPT and Gemini, SGD methods are employed to create successful generative AI based text-to-image creation models such as Midjourney, DALL-E, and Stable Diffusion, but SGD methods are also used to train DNNs to approximately solve scientific models such as partial differential equation (PDE) models from physics and biology and optimal control and stopping problems from engineering. It is known that the plain vanilla standard SGD method fails to converge even in the situation of several convex optimization problems if the learning rates are bounded away from zero. However, in many practical relevant training scenarios, often not the plain vanilla standard SGD method but instead adaptive SGD methods such as the RMSprop and the Adam optimizers, in which the learning rates are modified adaptively during the training process, are employed. This naturally rises the question whether such adaptive optimizers, in which the learning rates are modified adaptively during the training process, do converge in the situation of non-vanishing learning rates. In this work we answer this question negatively by proving that adaptive SGD methods such as the popular Adam optimizer fail to converge to any possible random limit point if the learning rates are asymptotically bounded away from zero. In our proof of this non-convergence result we establish suitable pathwise a priori bounds for a class of accelerated and adaptive SGD methods, which are also of independent interest.

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD

A Novel Convergence Analysis for Algorithms of the Adam Family

UAdam: Unified Adam-Type Algorithmic Framework for Nonconvex Optimization

UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization

Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration.

Convergence rates for the Adam optimizer

Provable Adaptivity of Adam under Non-uniform Smoothness

A modification of adaptive moment estimation (adam) for machine learning

On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond

On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions

Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates

Asymptotic and Non-Asymptotic Convergence of AdaGrad for Non-Convex Optimization via Novel Stopping Time-based Analysis

Adam$^+$: A Stochastic Method with Adaptive Variance Reduction

A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning

High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noise

Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning

Divergence Results and Convergence of a Variance Reduced Version of ADAM

Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case

SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients

A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings

A Bounded Scheduling Method for Adaptive Gradient Methods