Abstract:Momentum methods, such as heavy ball method (HB) and Nesterov’s accelerated gradient method (NAG), have been widely used in training neural networks by incorporating the history of gradients into the current updating process. In practice, they often provide improved performance over (stochastic) gradient descent (GD) with faster convergence. Despite their empirical success, a theoretical understanding of their accelerated convergence rates is still insufficient. Recently, some attempts have been made by analyzing the trajectories of gradient-based methods in an over-parameterized regime, where the number of the parameters is significantly larger than that of the training instances. However, the majority of existing theoretical work is mainly concerned with GD and the established convergence result of NAG is inferior to HB and GD, which fails to explain the practical success of NAG. In this paper, we take a step towards closing this gap by analyzing NAG in training a randomly initialized over-parameterized two-layer fully connected neural network with ReLU activation. Despite the fact that the objective function is non-convex and non-smooth, we show that NAG converges to a global minimum at a non-asymptotic linear rate (1−Θ(1/κ))t, where κ>1 is the condition number of a gram matrix and t is the number of the iterations. Compared to the convergence rate (1−Θ(1/κ))t of GD, our result provides theoretical guarantees for the acceleration of NAG in neural network training. Furthermore, our findings suggest that NAG and HB have similar convergence rate. Finally, extensive experiments on six benchmark datasets have been conducted to validate the correctness of our theoretical results.

Deterministic convergence of chaos injection-based gradient method for training feedforward neural networks.

Convergence of Chaos Injection-Based Batch Backpropagation Algorithm for Feedforward Neural Networks

Deterministic Convergence of Conjugate Gradient Method for Feedforward Neural Networks

Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Convergence of batch gradient algorithm for feedforward neural network training

Convergence of an online gradient method with inner-product penalty and adaptive momentum

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Deterministic Convergence of Wirtinger-Gradient Methods for Complex-Valued Neural Networks.

Convergence of Gradient Method for A Fully Recurrent Neural Network

Convergence of Batch Gradient Method Based on the Entropy Error Function for Feedforward Neural Networks

Fully Complex Conjugate Gradient-Based Neural Networks Using Wirtinger Calculus Framework: Deterministic Convergence and Its Application.

Convergence Analysis of Three Classes of Split-Complex Gradient Algorithms for Complex-Valued Recurrent Neural Networks

Convergence Analysis of Online Gradient Method for BP Neural Networks

Convergence of gradient method for double parallel feedforward neural network

Deterministic Convergence of Complex Mini-Batch Gradient Learning Algorithm for Fully Complex-Valued Neural Networks

Provable convergence of Nesterov’s accelerated gradient method for over-parameterized neural networks

Convergence of Online Gradient Method for Double Parallel Feedforward Neural Network

An Improved Analysis of Training Over-parameterized Deep Neural Networks

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Strong Convergence Of Gradient Methods For Bp Networks Training