Abstract:It is known that the standard stochastic gradient descent (SGD) optimization method, as well as accelerated and adaptive SGD optimization methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (as, for example, in the situation of constant learning rates). Numerical simulations often use human-tuned deterministic learning rate schedules or small constant learning rates. The default learning rate schedules for SGD optimization methods in machine learning implementation frameworks such as TensorFlow and Pytorch are constant learning rates. In this work we propose and study a learning-rate-adaptive approach for SGD optimization methods in which the learning rate is adjusted based on empirical estimates for the values of the objective function of the considered optimization problem (the function that one intends to minimize). In particular, we propose a learning-rate-adaptive variant of the Adam optimizer and implement it in case of several neural network learning problems, particularly, in the context of deep learning approximation methods for partial differential equations such as deep Kolmogorov methods, physics-informed neural networks, and deep Ritz methods. In each of the presented learning problems the proposed learning-rate-adaptive variant of the Adam optimizer faster reduces the value of the objective function than the Adam optimizer with the default learning rate. For a simple class of quadratic minimization problems we also rigorously prove that a learning-rate-adaptive variant of the SGD optimization method converges to the minimizer of the considered minimization problem. Our convergence proof is based on an analysis of the laws of invariant measures of the SGD method as well as on a more general convergence analysis for SGD with random but predictable learning rates which we develop in this work.

An Attempt of Applying the Lagrange-type 1-Step-ahead Numerical Differentiation Method to Optimize the SGD Algorithm in Deep Learning

Stochastic Gradient Descent in the Viewpoint of Graduated Optimization

Reducing Bias in Deep Learning Optimization: The RSGDM Approach

An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks

Gradient Convergence of Deep Learning-Based Numerical Methods for BSDEs

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes

A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference

Diagonalisation SGD: Fast & Convergent SGD for Non-Differentiable Models via Reparameterisation and Smoothing

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

On the Diffusion Approximation of Nonconvex Stochastic Gradient Descent

Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization

Stochastic Approximate Gradient Descent via the Langevin Algorithm

Efficient and stable SAV-based methods for gradient flows arising from deep learning

Aiming towards the minimizers: fast convergence of SGD for overparametrized problems

On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)

Optimal Adaptive and Accelerated Stochastic Gradient Descent

AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy

Learning-rate-free Momentum SGD with Reshuffling Converges in Nonsmooth Nonconvex Optimization

Asynchronous Stochastic Gradient Descent with Decoupled Backpropagation and Layer-Wise Updates

A modification of adaptive moment estimation (adam) for machine learning

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training