Abstract:As the training process of deep neural networks involves expensive computational cost, speeding up the convergence is of great importance. Nesterov’s accelerated gradient (NAG) is one of the most popular accelerated optimizers in the deep learning community, which often exhibits improved convergence performance over gradient descent (GD) in practice. However, theoretical investigations of NAG mainly focus on the convex setting. Since the optimization landscape of the neural network is non-convex, little is known about the convergence and acceleration of NAG. Nowadays, some works make progress towards understanding the convergence of NAG in training over-parameterized neural networks, where the number of the parameters exceeds that of the training instances. Nonetheless, previous studies are limited to the two-layer neural network, which are far from explaining the remarkable success of NAG in optimizing deep neural networks. In this paper, we investigate the convergence of NAG in training two architectures of deep linear networks: deep fully-connected linear neural networks and deep linear ResNets. Based on the over-parameterization regime, we first analyze the residual dynamics induced by the training trajectory of NAG for a deep fully-connected linear neural network under random Gaussian initialization. Our results show that NAG can converge to the global minimum at a (1-O(1/κ))t rate when the width is near-linear in the depth of the network, where t is the number of iterations and κ>1 is a constant depending on the condition number of the feature matrix. Compared to the (1-O(1/κ))t rate of GD, NAG achieves an acceleration over GD. For deep linear ResNets, we utilize the same analytical approach and obtain a similar convergence result, while the width requirement is independent of the depth. To the best of our knowledge, these are the first theoretical guarantees for the convergence and acceleration of NAG in training deep neural networks. Numerical results show the acceleration of NAG compared to GD in terms of iterations. In addition, we conduct experiments to evaluate the effect of the depth on the convergence rate of NAG, which validate our derived conditions of the width. We hope our results may shed light on understanding the optimization behavior of NAG for modern deep neural networks.

Analytical Gradient Method For Fundamental Invariant Neural Networks

Provable convergence of Nesterov’s accelerated gradient method for over-parameterized neural networks

Neural Green's Function Accelerated Iterative Methods for Solving Indefinite Boundary Value Problems

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks

CALCULATIONS OF TWO-DIMENSIONAL TRANSIENT POTENTIAL AND ITS GRADIENT BY COMBINING DIRECT METHOD AND HILBERT TRANSFORM

Fitting Potential Energy Surfaces With Fundamental Invariant Neural Network. Ii. Generating Fundamental Invariants For Molecular Systems With Up To Ten Atoms

A convergence analysis of Nesterov’s accelerated gradient method in training deep linear neural networks

Analytical Energy Gradient Evaluation in Relativistic and Nonrelativistic Density Functional Calculations

GPINN with Neural Tangent Kernel Technique for Nonlinear Two Point Boundary Value Problems

Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representations

Efficient and stable SAV-based methods for gradient flows arising from deep learning

Gradient-optimized physics-informed neural networks (GOPINNs): a deep learning method for solving the complex modified KdV equation

A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms

Input gradient annealing neural network for solving low-temperature Fokker-Planck equations

Preconditioned Nonlinear Conjugate Gradient Method for Real-time Interior-point Hyperelasticity

An Adaptive Gradient Neural Network to Solve Dynamic Linear Matrix Equations

A Hessian-Free Gradient Flow (HFGF) method for the optimisation of deep learning neural networks

A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspective

A Gradient Free Neural Network Framework Based on Universal Approximation Theorem

A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations

Refining Potential Energy Surface through Dynamical Properties via Differentiable Molecular Simulation