Abstract:Deep learning algorithms - typically consisting of a class of deep neural networks trained by a stochastic gradient descent (SGD) optimization method - are nowadays the key ingredients in many artificial intelligence (AI) systems and have revolutionized our ways of working and living in modern societies. For example, SGD methods are used to train powerful large language models (LLMs) such as versions of ChatGPT and Gemini, SGD methods are employed to create successful generative AI based text-to-image creation models such as Midjourney, DALL-E, and Stable Diffusion, but SGD methods are also used to train DNNs to approximately solve scientific models such as partial differential equation (PDE) models from physics and biology and optimal control and stopping problems from engineering. It is known that the plain vanilla standard SGD method fails to converge even in the situation of several convex optimization problems if the learning rates are bounded away from zero. However, in many practical relevant training scenarios, often not the plain vanilla standard SGD method but instead adaptive SGD methods such as the RMSprop and the Adam optimizers, in which the learning rates are modified adaptively during the training process, are employed. This naturally rises the question whether such adaptive optimizers, in which the learning rates are modified adaptively during the training process, do converge in the situation of non-vanishing learning rates. In this work we answer this question negatively by proving that adaptive SGD methods such as the popular Adam optimizer fail to converge to any possible random limit point if the learning rates are asymptotically bounded away from zero. In our proof of this non-convergence result we establish suitable pathwise a priori bounds for a class of accelerated and adaptive SGD methods, which are also of independent interest.

AdaDB: an Adaptive Gradient Method with Data-Dependent Bound.

Adaptive Gradient Methods with Dynamic Bound of Learning Rate.

AdaX: Adaptive Gradient Descent with Exponential Long Term Memory

AdaXod: a new adaptive and momental bound algorithm for training deep neural networks

An Adaptive and Momental Bound Method for Stochastic Learning

A New Adaptive Gradient Method with Gradient Decomposition

A fast adaptive algorithm for training deep neural networks

An Adaptive Mechanism to Achieve Learning Rate Dynamically

Adaptive Learning Rates with Maximum Variation Averaging.

ADADELTA: An Adaptive Learning Rate Method

ABNGrad: adaptive step size gradient descent for optimizing neural networks

Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates

AdamL: A fast adaptive gradient method incorporating loss function

Distributed Adaptive Subgradient Algorithms for Online Learning over Time-Varying Networks

AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods

CADA: Communication-Adaptive Distributed Adam

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods

An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent

Dynamic Batch Adaptation

Momentum is All You Need for Data-Driven Adaptive Optimization

ADOPT: Modified Adam Can Converge with Any $β_2$ with the Optimal Rate