Abstract:Momentum methods, such as heavy ball method (HB) and Nesterov’s accelerated gradient method (NAG), have been widely used in training neural networks by incorporating the history of gradients into the current updating process. In practice, they often provide improved performance over (stochastic) gradient descent (GD) with faster convergence. Despite their empirical success, a theoretical understanding of their accelerated convergence rates is still insufficient. Recently, some attempts have been made by analyzing the trajectories of gradient-based methods in an over-parameterized regime, where the number of the parameters is significantly larger than that of the training instances. However, the majority of existing theoretical work is mainly concerned with GD and the established convergence result of NAG is inferior to HB and GD, which fails to explain the practical success of NAG. In this paper, we take a step towards closing this gap by analyzing NAG in training a randomly initialized over-parameterized two-layer fully connected neural network with ReLU activation. Despite the fact that the objective function is non-convex and non-smooth, we show that NAG converges to a global minimum at a non-asymptotic linear rate (1−Θ(1/κ))t, where κ>1 is the condition number of a gram matrix and t is the number of the iterations. Compared to the convergence rate (1−Θ(1/κ))t of GD, our result provides theoretical guarantees for the acceleration of NAG in neural network training. Furthermore, our findings suggest that NAG and HB have similar convergence rate. Finally, extensive experiments on six benchmark datasets have been conducted to validate the correctness of our theoretical results.

Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks

Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Improving Deep Neural Networks' Training for Image Classification with Nonlinear Conjugate Gradient-Style Adaptive Momentum.

Provable convergence of Nesterov’s accelerated gradient method for over-parameterized neural networks

Convergence of Batch Gradient Method Based on the Entropy Error Function for Feedforward Neural Networks

Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization

Boundedness and Convergence of Batch Back-Propagation Algorithm with Penalty for Feedforward Neural Networks

Last-iterate convergence analysis of stochastic momentum methods for neural networks

Combining Conjugate Gradient and Momentum for Unconstrained Stochastic Optimization With Applications to Machine Learning

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Convergence rates of stochastic gradient method with independent sequences of step-size and momentum weight

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization

Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise

Stochastic normalized gradient descent with momentum for large-batch training

Novel Convergence Results of Adaptive Stochastic Gradient Descents

On the fast convergence of minibatch heavy ball momentum

Convergence of Gradient Algorithms for Nonconvex C1+α Cost Functions

A Recipe for Global Convergence Guarantee in Deep Neural Networks

A Convergent ADMM Framework for Efficient Neural Network Training