Abstract:Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to their success in practical problems, and there is some understanding of the role of stochastic gradient descent in this context. Momentum modifications of gradient descent such as Polyak's Heavy Ball method (HB) and Nesterov's method of accelerated gradients (NAG), are also widely adopted. In this work our focus is on understanding the role of momentum in the training of neural networks, concentrating on the common situation in which the momentum contribution is fixed at each step of the algorithm. To expose the ideas simply we work in the deterministic setting. Our approach is to derive continuous time approximations of the discrete algorithms; these continuous time approximations provide insights into the mechanisms at play within the discrete algorithms. We prove three such approximations. Firstly we show that standard implementations of fixed momentum methods approximate a time-rescaled gradient descent flow, asymptotically as the learning rate shrinks to zero; this result does not distinguish momentum methods from pure gradient descent, in the limit of vanishing learning rate. We then proceed to prove two results aimed at understanding the observed practical advantages of fixed momentum methods over gradient descent. We achieve this by proving approximations to continuous time limits in which the small but fixed learning rate appears as a parameter. Furthermore in a third result we show that the momentum methods admit an exponentially attractive invariant manifold on which the dynamics reduces, approximately, to a gradient flow with respect to a modified loss function.

Dynamic behavior for a gradient algorithm with energy and momentum

Adaptive Preconditioned Gradient Descent with Energy

A Globally Convergent Gradient Method with Momentum

Dyna: A Method of Momentum for Stochastic Optimization

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

Convergence of Gradient Algorithms for Nonconvex C1+α Cost Functions

Continuous Time Analysis of Momentum Methods

When is Momentum Extragradient Optimal? A Polynomial-Based Analysis

A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization

mPage: Probabilistic Gradient Estimator With Momentum for Non-Convex Optimization

Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima

Optimal Convergence Rates for Damped Inertial Gradient Dynamics with Flat Geometries

Non-Convex Stochastic Composite Optimization with Polyak Momentum

An SDE Perspective on Stochastic Inertial Gradient Dynamics with Time-Dependent Viscosity and Geometric Damping

Accelerated Alternating Direction Method of Multipliers Gradient Tracking for Distributed Optimization

ADINE: An Adaptive Momentum Method for Stochastic Gradient Descent

A Particle Gradient Evolutionary Algorithm Based On Statistical Mechanics And Convergence Analysis

Does Momentum Change the Implicit Regularization on Separable Data?

A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms

An Accelerated Distributed Stochastic Gradient Method with Momentum

Accelerated Sparse Recovery Via Gradient Descent with Nonlinear Conjugate Gradient Momentum