Abstract:This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved.

Convergence of approximated gradient method for Elman network

Convergence of Gradient Method for Elman Networks

Convergence Analysis of Regularized Elman Neural Networks under Relaxed Conditions

Convergence Analysis of an Improved Extreme Learning Machine Based on Gradient Descent Method

Weak and strong convergence analysis of Elman neural networks via weight decay regularization

Approximation and Gradient Descent Training with Neural Networks

Convergence of Gradient Method for A Fully Recurrent Neural Network

Convergence of gradient method for double parallel feedforward neural network

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

On the convergence of gradient descent for two layer neural networks

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Convergence of batch gradient algorithm for feedforward neural network training

Approximation Property of the Modified Elman Network

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Quantum Gate Elman Neural Network and Its Quantized Extended Gradient Back-propagation Training Algorithm: Quantum Gate Elman Neural Network and Its Quantized Extended Gradient Back-propagation Training Algorithm

An Algorithm Of Supervised Learning For Elman Neural Network

Convergence of online gradient method for pi-sigma neural networks

On Convergence of Training Loss Without Reaching Stationary Points