Abstract:This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved.

Convergence of Online Gradient Method with a Penalty Term for BP Neural Network with Stochastic Inputs

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Boundedness and Convergence of Batch Back-Propagation Algorithm with Penalty for Feedforward Neural Networks

Global Convergence of Online Bp Training with Dynamic Learning Rate

When does online BP training converge?

A New Convergence Property of Online BP Learning

New and Rapidly Converging BP Neural Network Algorithm

An Algorithm for Fast Convergence of Back Propagation by Enlarging Error

An optimizing BP neural network algorithm based on genetic algorithm

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Stochastic Gradient Descent Method with Convex Penalty for Ill-Posed Problems in Banach Spaces

Improving Steepest Descent Method by Learning Rate Annealing and Momentum in Neural Network

Back Propagation With Randomized Cost Function For Training Neural Networks

An Effective Backpropagation Algorithm for Optimizing BP Neural Network Based on Rough Set and Modified Genetic Algorithm

On Penalty-based Bilevel Gradient Descent Method

Training Feed-forward Neural Networks Using the Gradient Descent Method with the Optimal Stepsize

Gradient Convergence of Deep Learning-Based Numerical Methods for BSDEs

A New Global Optimization Algorithm for Training Feedforward Neural Networks and Its Application

Convergence of Batch Gradient Method Based on the Entropy Error Function for Feedforward Neural Networks