Abstract:In recent years, stochastic gradient descent (SGD) based techniques has become the standard tools for training neural networks. However, formal theoretical understanding of why SGD can train neural networks in practice is largely missing. In this paper, we make progress on understanding this mystery by providing a convergence analysis for SGD on a rich subset of two-layer feedforward networks with ReLU activations. This subset is characterized by a special structure called "identity mapping". We prove that, if input follows from Gaussian distribution, with standard $O(1/\sqrt{d})$ initialization of the weights, SGD converges to the global minimum in polynomial number of steps. Unlike normal vanilla networks, the "identity mapping" makes our network asymmetric and thus the global minimum is unique. To complement our theory, we are also able to show experimentally that multi-layer networks with this mapping have better performance compared with normal vanilla networks. Our convergence theorem differs from traditional non-convex optimization techniques. We show that SGD converges to optimal in "two phases": In phase I, the gradient points to the wrong direction, however, a potential function $g$ gradually decreases. Then in phase II, SGD enters a nice one point convex region and converges. We also show that the identity mapping is necessary for convergence, as it moves the initial point to a better place for optimization. Experiment verifies our claims.

Convergence Analysis of Batch Gradient Algorithm for Three Classes of Sigma-Pi Neural Networks

Convergence of online gradient method for pi-sigma neural networks

Convergence of gradient descent algorithm for Pi-sigma neural networks

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Convergence of online gradient methods for Pi-Sigma neural network with a penalty term

Convergence of Batch Gradient Learning Algorithm with Smoothing L1/2 Regularization for Sigma–Pi–Sigma Neural Networks

Convergence Analysis of Three Classes of Split-Complex Gradient Algorithms for Complex-Valued Recurrent Neural Networks

Convergence of batch gradient algorithm for feedforward neural network training

Convergence Analysis of Online Gradient Method for BP Neural Networks

Training Pi-Sigma Network by Online Gradient Algorithm with Penalty for Small Weight Update.

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

A new Sigma-Pi-Sigma neural network based on $ L_1 $ and $ L_2 $ regularization and applications

Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks

Numerical Analysis for Convergence of a Sample-Wise Backpropagation Method for Training Stochastic Neural Networks

A Class of Gradient Algorithms with Variable Learning Rates and Convergence Analysis for Feedforward Neural Networks Training

A Unified Analysis of Stochastic Momentum Methods for Deep Learning

The Convergence Analysis of SpikeProp Algorithm with Smoothing L 1 ∕ 2 Regularization

A Unified Analysis of Stochastic Momentum Methods for Deep Learning.

Deterministic convergence of chaos injection-based gradient method for training feedforward neural networks.

Deterministic Convergence of Conjugate Gradient Method for Feedforward Neural Networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation