Abstract:The backpropogation (BP) neural networks have been widely applied in scientific research and engineering. The success of the application, however, relies upon the convergence of the training procedure involved in the neural network learning. We settle down the convergence analysis issue through proving two fundamental theorems on the convergence of the online BP training procedure. One theorem claims that under mild conditions, the gradient sequence of the error function will converge to zero (the weak convergence), and another theorem concludes the convergence of the weight sequence defined by the procedure to a fixed value at which the error function attains its minimum (the strong convergence). The weak convergence theorem sharpens and generalizes the existing convergence analysis conducted before, while the strong convergence theorem provides new analysis results on convergence of the online BP training procedure. The results obtained reveal that with any analytic sigmoid activation function, the online BP training procedure is always convergent, which then underlies successful application of the BP neural networks.

Convergence of Gradient Algorithm for BP Neural Network with Multiple Output Units

Strong Convergence Of Gradient Methods For Bp Networks Training

Convergence Analysis of Online Gradient Method for BP Neural Networks

Convergence of BP Algorithm for Training MLP with Linear Output

Convergence of batch gradient algorithm for feedforward neural network training

Convergence of Online Gradient Method with a Penalty Term for BP Neural Network with Stochastic Inputs

Convergence of Gradient Method with Momentum for Back-Propagation Neural Networks

When does online BP training converge?

Convergence Analysis of BP Neural Networks Via Sparse Response Regularization

BP Neural Network Algorithm Improvements and Convergence Analysis

Convergence of BP Algorithm for Product Unit Neural Networks with Exponential Weights.

Relaxed Conditions for Convergence of Batch BPAP for Feedforward Neural Networks.

The Improvements of Bp Neural Network Learning Algorithm

A Novel Conjugate Gradient Method with Generalized Armijo Search for Efficient Training of Feedforward Neural Networks

Convergence of Batch Gradient Learning with Smoothing Regularization and Adaptive Momentum for Neural Networks

Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks

Convergence of BP Algorithm with Variable Learning Rates for FNN Training

Convergence of gradient method for double parallel feedforward neural network

A New Convergence Property of Online BP Learning

Convergence of Gradient Method with Momentum for Two-Layer Feedforward Neural Networks

Convergence of batch BP algorithm with penalty for FNN training