Abstract:Momentum methods, such as heavy ball method (HB) and Nesterov’s accelerated gradient method (NAG), have been widely used in training neural networks by incorporating the history of gradients into the current updating process. In practice, they often provide improved performance over (stochastic) gradient descent (GD) with faster convergence. Despite their empirical success, a theoretical understanding of their accelerated convergence rates is still insufficient. Recently, some attempts have been made by analyzing the trajectories of gradient-based methods in an over-parameterized regime, where the number of the parameters is significantly larger than that of the training instances. However, the majority of existing theoretical work is mainly concerned with GD and the established convergence result of NAG is inferior to HB and GD, which fails to explain the practical success of NAG. In this paper, we take a step towards closing this gap by analyzing NAG in training a randomly initialized over-parameterized two-layer fully connected neural network with ReLU activation. Despite the fact that the objective function is non-convex and non-smooth, we show that NAG converges to a global minimum at a non-asymptotic linear rate (1−Θ(1/κ))t, where κ>1 is the condition number of a gram matrix and t is the number of the iterations. Compared to the convergence rate (1−Θ(1/κ))t of GD, our result provides theoretical guarantees for the acceleration of NAG in neural network training. Furthermore, our findings suggest that NAG and HB have similar convergence rate. Finally, extensive experiments on six benchmark datasets have been conducted to validate the correctness of our theoretical results.

The Research on the Relation of Self-Learning Ratio and the Convergence Speed in BP Networks

Improved BP algorithm for neural network and its application in welding process control

New and Rapidly Converging BP Neural Network Algorithm

A Study of Application of An Improved PSO Algorithm in BP Network

The Improved Rapid Convergence Algorithm of the Connecting Rights in the BP Network

Research and Application on BP Neural Network Algorithm

A Theoretical View of Linear Backpropagation and Its Convergence

A New Convergence Property of Online BP Learning

Optimal parameters selection of back propagation algorithm in the feedforward neural network

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

An optimizing BP neural network algorithm based on genetic algorithm

Research and application on improved BP neural network algorithm

Optimization Study of BP Neural Network Based on Genetic Algorithm

Brief Introduction of Back Propagation (BP) Neural Network Algorithm and Its Improvement

Convergence rates of stochastic gradient method with independent sequences of step-size and momentum weight

Improving Steepest Descent Method by Learning Rate Annealing and Momentum in Neural Network

On Learning Rates and Schrödinger Operators

BiSLS/SPS: Auto-tune Step Sizes for Stable Bi-level Optimization

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Provable convergence of Nesterov’s accelerated gradient method for over-parameterized neural networks

A constrained optimization method based on BP neural network