Abstract:Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of two stochastic momentum methods, namely stochastic heavy-ball method and the stochastic variant of Nesterov's accelerated gradient method. We hope that the basic convergence results developed in this paper can serve the reference to the convergence of stochastic momentum methods and also serve the baselines for comparison in future development of stochastic momentum methods. The novelty of convergence analysis presented in this paper is a unified framework, revealing more insights about the similarities and differences between different stochastic momentum methods and stochastic gradient method. The unified framework exhibits a continuous change from the gradient method to Nesterov's accelerated gradient method and finally the heavy-ball method incurred by a free parameter, which can help explain a similar change observed in the testing error convergence behavior for deep learning. Furthermore, our empirical results for optimizing deep neural networks demonstrate that the stochastic variant of Nesterov's accelerated gradient method achieves a good tradeoff (between speed of convergence in training error and robustness of convergence in testing error) among the three stochastic methods.

Boundedness and Convergence of Online Gradient Method with Penalty and Momentum

Convergence and monotonicity of an online gradient method with penalty for neural networks

Boundedness and Convergence of Online Gradient Method with Penalty for Linear Output Feedforward Neural Networks

Convergence of an online gradient method with inner-product penalty and adaptive momentum

Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Boundedness and Convergence of Split-Complex Back-Propagation Algorithm with Momentum and Penalty

Convergence analysis of a back-propagation algorithm with adaptive momentum

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Convergence of Online Gradient Method with a Penalty Term for BP Neural Network with Stochastic Inputs

Strong Convergence Of Gradient Methods For Bp Networks Training

Convergence of Gradient Method with Momentum for Two-Layer Feedforward Neural Networks

Boundedness and Convergence of Batch Back-Propagation Algorithm with Penalty for Feedforward Neural Networks

Convergence of A Gradient Algorithm with Penalty for Training Two-Layer Neural Networks

Online Gradient Methods with a Punishing Term for Neural Networks

Convergence Analysis of Online Gradient Method for BP Neural Networks

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

Convergence of Online Gradient Method for Double Parallel Feedforward Neural Network

Relaxed Conditions for Convergence of Batch BPAP for Feedforward Neural Networks.

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization