Abstract:Deep learning algorithms - typically consisting of a class of deep neural networks trained by a stochastic gradient descent (SGD) optimization method - are nowadays the key ingredients in many artificial intelligence (AI) systems and have revolutionized our ways of working and living in modern societies. For example, SGD methods are used to train powerful large language models (LLMs) such as versions of ChatGPT and Gemini, SGD methods are employed to create successful generative AI based text-to-image creation models such as Midjourney, DALL-E, and Stable Diffusion, but SGD methods are also used to train DNNs to approximately solve scientific models such as partial differential equation (PDE) models from physics and biology and optimal control and stopping problems from engineering. It is known that the plain vanilla standard SGD method fails to converge even in the situation of several convex optimization problems if the learning rates are bounded away from zero. However, in many practical relevant training scenarios, often not the plain vanilla standard SGD method but instead adaptive SGD methods such as the RMSprop and the Adam optimizers, in which the learning rates are modified adaptively during the training process, are employed. This naturally rises the question whether such adaptive optimizers, in which the learning rates are modified adaptively during the training process, do converge in the situation of non-vanishing learning rates. In this work we answer this question negatively by proving that adaptive SGD methods such as the popular Adam optimizer fail to converge to any possible random limit point if the learning rates are asymptotically bounded away from zero. In our proof of this non-convergence result we establish suitable pathwise a priori bounds for a class of accelerated and adaptive SGD methods, which are also of independent interest.

Global Convergence of GHA Learning Algorithm with Nonzero-Approaching Adaptive Learning Rates

Global convergence of Oja's PCA learning algorithm with a non-zero-approaching adaptive learning rate

Global Convergence of a PCA Learning Algorithm with a Constant Learning Rate

Global convergence analysis of a PCA learning algorithm

An adaptive learning algorithm for principal component analysis

A Globally Convergent MC Algorithm With an Adaptive Learning Rate.

Dynamics of Generalized PCA and MCA Learning Algorithms.

A Robust And Globally Convergent Pca Learning Algorithm

Convergence analysis of Chauvin’s PCA learning algorithm with a constant learning rate

A Globally Convergent Learning Algorithm for Pca Neural Networks

Global Convergence of an Adaptive Minor Component Extraction Algorithm

Convergence analysis of a deterministic discrete time system of Oja's PCA learning algorithm.

Global Convergence Analysis of a Discrete Time Nonnegative ICA Algorithm

Transient Performance Analysis of Zero-Attracting Gaussian Kernel LMS Algorithm with Pre-Tuned Dictionary.

Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates

Global Convergence Analysis of a Self-Stabilizing MCA Learning Algorithm.

A Polak-Ribière-Polyak Conjugate Gradient-Based Neuro-Fuzzy Network and Its Convergence

Non-Divergence Of Stochastic Discrete Time Algorithms For Pca Neural Networks

Concaving Space Algorithm For Neural Network

Local Convergence of Gradient Descent-Ascent for Training Generative Adversarial Networks

Fixed-point Generalized Maximum Correntropy: Convergence Analysis and Convex Combination Algorithms