Abstract:Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of two stochastic momentum methods, namely stochastic heavy-ball method and the stochastic variant of Nesterov's accelerated gradient method. We hope that the basic convergence results developed in this paper can serve the reference to the convergence of stochastic momentum methods and also serve the baselines for comparison in future development of stochastic momentum methods. The novelty of convergence analysis presented in this paper is a unified framework, revealing more insights about the similarities and differences between different stochastic momentum methods and stochastic gradient method. The unified framework exhibits a continuous change from the gradient method to Nesterov's accelerated gradient method and finally the heavy-ball method incurred by a free parameter, which can help explain a similar change observed in the testing error convergence behavior for deep learning. Furthermore, our empirical results for optimizing deep neural networks demonstrate that the stochastic variant of Nesterov's accelerated gradient method achieves a good tradeoff (between speed of convergence in training error and robustness of convergence in testing error) among the three stochastic methods.

New Kind of Convergence Algorithm about Wavelet Neural Network Based on Spectrum Analysis

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

An Algorithm Of Wavelet Network Learning From Noisy Data

Construction of a New Adaptive Wavelet Network and Its Learning Algorithm

On Convergence of Training Loss Without Reaching Stationary Points

Wavelet J-Net: A Frequency Perspective on Convolutional Neural Networks

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Concaving Space Algorithm For Neural Network

Multiresolution Algorithm of Wavelet Network Learning from Absent Training Data

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Anomaly Detection of Network Traffic Based on Wavelet Packet

Advancing Graph Convolutional Networks via General Spectral Wavelets

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Design And Application Of Wavelet Networks Based On Immune Algorithm

A HYBRID LEARNING ALGORITHM BASED ON LEVENBERGMARQUARDT ALGORITHM AND LEAST SQUARES TECHNIQUES FOR WAVELET NETWORKS

A new method for general work piece recognition based on Neural Network

Genetic Algorithms Based on Wavelet Transform for Resolving Simulated Overlapped Spectra

A Pseudoinverse Incremental Algorithm For Fast Training Deep Neural Networks With Application To Spectra Pattern Recognition

Convergence Analysis of the Modified Frequency-Domain Block LMS Algorithm with Guaranteed Optimal Steady State Performance

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Wavelet Neural Networks for Adaptive Equalization