Abstract:Fractional-order derivatives have the potential to improve the performance of backpropagation (BP) neural networks. Several studies have found that the fractional-order gradient learning methods may not converge to real extreme points. The truncation and the modification of the fractional-order derivative are applied to guarantee convergence to the real extreme point. Nonetheless, the real convergence ability is based on the assumption that the algorithm is convergent, which limits the practicality of the algorithm. In this article, a novel truncated fractional-order BP neural network (TFO-BPNN) and a novel hybrid TFO-BPNN (HTFO-BPNN) are designed to solve the above problem. First, to avoid overfitting, a squared regularization term is introduced into the fractional-order BP neural network. Second, a novel dual cross-entropy cost function is proposed and employed as a loss function for the two neural networks. The penalty parameter helps to adjust the effect of the penalty term and further alleviates the gradient vanishing problem. In terms of convergence, the convergence ability of the two proposed neural networks is first proved. Then, the convergence ability to the real extreme point is further analyzed theoretically. Finally, the simulation results effectively illustrate the feasibility, high accuracy, and good generalization ability of the proposed neural networks. Comparative studies among the proposed neural networks and some related methods further substantiate the superiority of the TFO-BPNN and the HTFO-BPNN.

Convergence analysis of fully complex backpropagation algorithm based on Wirtinger calculus.

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

A Complex Hebbian-type Learning Algorithm with Application to Complex Adaptive ⅡR Filtering

Convergence Analysis of Novel Fractional-Order Backpropagation Neural Networks With Regularization Terms

Complex-Valued Wavelet Network

Complex-Valued Recurrent Neural Network with IIR Neuron Model: Training and Applications

Convergence analysis of inexact proximal point algorithms on Hadamard manifolds

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

A L-BFGS Based Learning Algorithm for Complex-Valued Feedforward Neural Networks

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Real-Valued Backpropagation is Unsuitable for Complex-Valued Neural Networks

A New Convergence Property of Online BP Learning

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Convergence Analysis for Over-Parameterized Deep Learning

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Convergence Analysis of Deep Residual Networks

Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer

A Theoretical View of Linear Backpropagation and Its Convergence

Convergent dynamics analysis of neural networks: A decomposition approach

Weak and strong convergence analysis of Elman neural networks via weight decay regularization

Optimal parameters selection of back propagation algorithm in the feedforward neural network