Abstract:Deep learning algorithms - typically consisting of a class of deep neural networks trained by a stochastic gradient descent (SGD) optimization method - are nowadays the key ingredients in many artificial intelligence (AI) systems and have revolutionized our ways of working and living in modern societies. For example, SGD methods are used to train powerful large language models (LLMs) such as versions of ChatGPT and Gemini, SGD methods are employed to create successful generative AI based text-to-image creation models such as Midjourney, DALL-E, and Stable Diffusion, but SGD methods are also used to train DNNs to approximately solve scientific models such as partial differential equation (PDE) models from physics and biology and optimal control and stopping problems from engineering. It is known that the plain vanilla standard SGD method fails to converge even in the situation of several convex optimization problems if the learning rates are bounded away from zero. However, in many practical relevant training scenarios, often not the plain vanilla standard SGD method but instead adaptive SGD methods such as the RMSprop and the Adam optimizers, in which the learning rates are modified adaptively during the training process, are employed. This naturally rises the question whether such adaptive optimizers, in which the learning rates are modified adaptively during the training process, do converge in the situation of non-vanishing learning rates. In this work we answer this question negatively by proving that adaptive SGD methods such as the popular Adam optimizer fail to converge to any possible random limit point if the learning rates are asymptotically bounded away from zero. In our proof of this non-convergence result we establish suitable pathwise a priori bounds for a class of accelerated and adaptive SGD methods, which are also of independent interest.

Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks

Conjugate-gradient-based Adam for stochastic optimization and its application to deep learning

Adaptive Stochastic Conjugate Gradient Optimization for Backpropagation Neural Networks

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization

A Nonmonotone Scaled Conjugate Gradient Algorithm for Large-Scale Unconstrained Optimization.

The Conjugate Gradient Method with neural network control

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes

Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutions

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

On Faster Convergence of Scaled Sign Gradient Descent

Gradient Methods with Online Scaling

Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

Explicit and Implicit Graduated Optimization in Deep Neural Networks

Convergence of Stochastic Gradient Descent in Deep Neural Network

Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods