Abstract:It is known that the standard stochastic gradient descent (SGD) optimization method, as well as accelerated and adaptive SGD optimization methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (as, for example, in the situation of constant learning rates). Numerical simulations often use human-tuned deterministic learning rate schedules or small constant learning rates. The default learning rate schedules for SGD optimization methods in machine learning implementation frameworks such as TensorFlow and Pytorch are constant learning rates. In this work we propose and study a learning-rate-adaptive approach for SGD optimization methods in which the learning rate is adjusted based on empirical estimates for the values of the objective function of the considered optimization problem (the function that one intends to minimize). In particular, we propose a learning-rate-adaptive variant of the Adam optimizer and implement it in case of several neural network learning problems, particularly, in the context of deep learning approximation methods for partial differential equations such as deep Kolmogorov methods, physics-informed neural networks, and deep Ritz methods. In each of the presented learning problems the proposed learning-rate-adaptive variant of the Adam optimizer faster reduces the value of the objective function than the Adam optimizer with the default learning rate. For a simple class of quadratic minimization problems we also rigorously prove that a learning-rate-adaptive variant of the SGD optimization method converges to the minimizer of the considered minimization problem. Our convergence proof is based on an analysis of the laws of invariant measures of the SGD method as well as on a more general convergence analysis for SGD with random but predictable learning rates which we develop in this work.

Conjugate-gradient-based Adam for stochastic optimization and its application to deep learning

StochGradAdam: Accelerating Neural Networks Training with Stochastic Gradient Sampling

Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning

On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions

Adam: A Method for Stochastic Optimization

Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks

A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning

Adam$^+$: A Stochastic Method with Adaptive Variance Reduction

Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks

Variational Stochastic Gradient Descent for Deep Neural Networks

Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

Convergence rates for the Adam optimizer

Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees

UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization

A modification of adaptive moment estimation (adam) for machine learning

ADOPT: Modified Adam Can Converge with Any $β_2$ with the Optimal Rate

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

AMAdam: adaptive modifier of Adam method

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD

An Adaptive Gradient Method with Differentiation Element in Deep Neural Networks

Adaptive Stochastic Conjugate Gradient Optimization for Backpropagation Neural Networks