Abstract:Fractional-order derivatives have the potential to improve the performance of backpropagation (BP) neural networks. Several studies have found that the fractional-order gradient learning methods may not converge to real extreme points. The truncation and the modification of the fractional-order derivative are applied to guarantee convergence to the real extreme point. Nonetheless, the real convergence ability is based on the assumption that the algorithm is convergent, which limits the practicality of the algorithm. In this article, a novel truncated fractional-order BP neural network (TFO-BPNN) and a novel hybrid TFO-BPNN (HTFO-BPNN) are designed to solve the above problem. First, to avoid overfitting, a squared regularization term is introduced into the fractional-order BP neural network. Second, a novel dual cross-entropy cost function is proposed and employed as a loss function for the two neural networks. The penalty parameter helps to adjust the effect of the penalty term and further alleviates the gradient vanishing problem. In terms of convergence, the convergence ability of the two proposed neural networks is first proved. Then, the convergence ability to the real extreme point is further analyzed theoretically. Finally, the simulation results effectively illustrate the feasibility, high accuracy, and good generalization ability of the proposed neural networks. Comparative studies among the proposed neural networks and some related methods further substantiate the superiority of the TFO-BPNN and the HTFO-BPNN.

Convergence of batch BP algorithm with penalty for FNN training

Boundedness and Convergence of Batch Back-Propagation Algorithm with Penalty for Feedforward Neural Networks

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

A Learning Algorithm for Solving the Over-Fitting Problem in BPANN

Convergence Analysis of Novel Fractional-Order Backpropagation Neural Networks With Regularization Terms

When does online BP training converge?

Optimal parameters selection of back propagation algorithm in the feedforward neural network

An Algorithm for Fast Convergence of Back Propagation by Enlarging Error

Convergence of Batch Gradient Method Based on the Entropy Error Function for Feedforward Neural Networks

New and Rapidly Converging BP Neural Network Algorithm

Global Convergence of Online Bp Training with Dynamic Learning Rate

Boundedness and Convergence of Mini-batch Gradient Method with Cyclic Dropconnect and Penalty

A New Convergence Property of Online BP Learning

A New Global Optimization Algorithm for Training Feedforward Neural Networks and Its Application

Back Propagation With Randomized Cost Function For Training Neural Networks

The Improved Rapid Convergence Algorithm of the Connecting Rights in the BP Network

Research and Application on BP Neural Network Algorithm

Brief Introduction of Back Propagation (BP) Neural Network Algorithm and Its Improvement

An Advanced NN Learning Algorithm Based on DFP Method

An optimizing BP neural network algorithm based on genetic algorithm

Fast Learning Algorithms for Feedforward Neural Networks