Abstract:Stochastic gradient descent (SGD) optimization methods are nowadays the method of choice for the training of deep neural networks (DNNs) in artificial intelligence systems. In practically relevant training problems, usually not the plain vanilla standard SGD method is the employed optimization scheme but instead suitably accelerated and adaptive SGD optimization methods are applied. As of today, maybe the most popular variant of such accelerated and adaptive SGD optimization methods is the famous Adam optimizer proposed by Kingma & Ba in 2014. Despite the popularity of the Adam optimizer in implementations, it remained an open problem of research to provide a convergence analysis for the Adam optimizer even in the situation of simple quadratic stochastic optimization problems where the objective function (the function one intends to minimize) is strongly convex. In this work we solve this problem by establishing optimal convergence rates for the Adam optimizer for a large class of stochastic optimization problems, in particular, covering simple quadratic stochastic optimization problems. The key ingredient of our convergence analysis is a new vector field function which we propose to refer to as the Adam vector field. This Adam vector field accurately describes the macroscopic behaviour of the Adam optimization process but differs from the negative gradient of the objective function (the function we intend to minimize) of the considered stochastic optimization problem. In particular, our convergence analysis reveals that the Adam optimizer does typically not converge to critical points of the objective function (zeros of the gradient of the objective function) of the considered optimization problem but converges with rates to zeros of this Adam vector field.

A Randomized Block-Coordinate Adam online learning optimization algorithm

Randomized Block-Coordinate Adaptive Algorithms for Nonconvex Optimization Problems

An Improved Adam Optimization Algorithm Combining Adaptive Coefficients and Composite Gradients Based on Randomized Block Coordinate Descent

Online Learning for DNN Training: A Stochastic Block Adaptive Gradient Algorithm

Adam: A Method for Stochastic Optimization

ABNGrad: adaptive step size gradient descent for optimizing neural networks

Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise

An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent

Reducing Bias in Deep Learning Optimization: The RSGDM Approach

A modification of adaptive moment estimation (adam) for machine learning

Convergence rates for the Adam optimizer

Adaptive Learning Rates with Maximum Variation Averaging.

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD

Adaptive Gradient Methods with Dynamic Bound of Learning Rate.

Byzantine-robust decentralized stochastic optimization with stochastic gradient noise-independent learning error

CADA: Communication-Adaptive Distributed Adam

AdaX: Adaptive Gradient Descent with Exponential Long Term Memory

UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization

Normalized Direction-preserving Adam.

Randomized Block Subgradient Methods for Convex Nonsmooth and Stochastic Optimization

Randomized block coordinate descent method for linear ill-posed problems