Abstract:Momentum methods, such as heavy ball method (HB) and Nesterov’s accelerated gradient method (NAG), have been widely used in training neural networks by incorporating the history of gradients into the current updating process. In practice, they often provide improved performance over (stochastic) gradient descent (GD) with faster convergence. Despite their empirical success, a theoretical understanding of their accelerated convergence rates is still insufficient. Recently, some attempts have been made by analyzing the trajectories of gradient-based methods in an over-parameterized regime, where the number of the parameters is significantly larger than that of the training instances. However, the majority of existing theoretical work is mainly concerned with GD and the established convergence result of NAG is inferior to HB and GD, which fails to explain the practical success of NAG. In this paper, we take a step towards closing this gap by analyzing NAG in training a randomly initialized over-parameterized two-layer fully connected neural network with ReLU activation. Despite the fact that the objective function is non-convex and non-smooth, we show that NAG converges to a global minimum at a non-asymptotic linear rate (1−Θ(1/κ))t, where κ>1 is the condition number of a gram matrix and t is the number of the iterations. Compared to the convergence rate (1−Θ(1/κ))t of GD, our result provides theoretical guarantees for the acceleration of NAG in neural network training. Furthermore, our findings suggest that NAG and HB have similar convergence rate. Finally, extensive experiments on six benchmark datasets have been conducted to validate the correctness of our theoretical results.

Optimisation of Overparametrized Sum-Product Networks

The Sum-Product Theorem: A Foundation for Learning Tractable Models

Deep Compression of Sum-Product Networks on Tensor Networks

Top-Down Bayesian Posterior Sampling for Sum-Product Networks

Training Overparametrized Neural Networks in Sublinear Time

On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions

Exploiting Problem Structure in Deep Declarative Networks: Two Case Studies

The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent

Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space

Sum-Product-Set Networks: Deep Tractable Models for Tree-Structured Graphs

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks

Provable convergence of Nesterov’s accelerated gradient method for over-parameterized neural networks

Tractable Probabilistic Graph Representation Learning with Graph-Induced Sum-Product Networks

Deep Parameter Optimisation

A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks

The Deep Parametric PDE Method: Application to Option Pricing

Interpolatron: Interpolation or Extrapolation Schemes to Accelerate Optimization for Deep Neural Networks.

Hyperparameter Optimization with Neural Network Pruning

Generalization and Expressivity for Deep Nets