Finite Convergence of On-line BP Neural Networks with Linearly Separable Training Patterns

Zhi-qiong SHAO,Wei WU,Jie YANG

DOI: https://doi.org/10.3770/j.issn.2095-2651.2006.03.004

2006-01-01

Abstract:In this paper we prove a finite convergence of online BP algorithms for nonlinear feedforward neural networks when the training patterns are linearly separable.

What problem does this paper attempt to address?

Convergence of online gradient methods for continuous perceptrons with linearly separable training patterns

Wei Wu,Zhiqiong Shao

DOI: https://doi.org/10.1016/S0893-9659(03)90086-3

IF: 4.294

2003-01-01

Applied Mathematics Letters

Abstract:In this paper, we prove that the online gradient method for continuous perceptrons converges in finite steps when the training patterns are linearly separable.
A New Convergence Property of Online BP Learning

Rui Zhang,Le Yang,Wei Wang

DOI: https://doi.org/10.1109/icist.2011.5765243

2011-01-01

Abstract:The feedforward neural networks trained with the online backpropagation (BP) learning algorithm have been widely studied in various areas of scientific research and engineering applications. In this paper we further study the convergence property of the online BP learning algorithm. Unlike the existing convergence analysis mainly focusing on the convergence of the gradient sequence of the error functions, we prove a convergence theorem for the sequence of the error functions itself.
When does online BP training converge?

Zong-Ben Xu,Rui Zhang,Wen-Feng Jing

DOI: https://doi.org/10.1109/TNN.2009.2025946

2009-01-01

Abstract:The backpropogation (BP) neural networks have been widely applied in scientific research and engineering. The success of the application, however, relies upon the convergence of the training procedure involved in the neural network learning. We settle down the convergence analysis issue through proving two fundamental theorems on the convergence of the online BP training procedure. One theorem claims that under mild conditions, the gradient sequence of the error function will converge to zero (the weak convergence), and another theorem concludes the convergence of the weight sequence defined by the procedure to a fixed value at which the error function attains its minimum (the strong convergence). The weak convergence theorem sharpens and generalizes the existing convergence analysis conducted before, while the strong convergence theorem provides new analysis results on convergence of the online BP training procedure. The results obtained reveal that with any analytic sigmoid activation function, the online BP training procedure is always convergent, which then underlies successful application of the BP neural networks.
Convergence of BP Algorithm for Training MLP with Linear Output

Shao Hongmei,Wu Wei,Liu Wenbin

2007-01-01

Abstract:The capability of multilayer perceptrons (MLPs) for approximating continuous func- tions with arbitrary accuracy has been demonstrated in the past decades. Back propa- gation (BP) algorithm is the most popular learning algorithm for training of MLPs. In this paper, a simple iteration formula is used to select the learning rate for each cycle of training procedure, and a convergence result is presented for the BP algorithm for training MLP with a hidden layer and a linear output unit. The monotonicity of the error function is also guaranteed during the training iteration.
Convergence Analysis of Online Gradient Method for BP Neural Networks

Wei Wu,Jian Wang,Mingsong Cheng,Zhengxue Li

DOI: https://doi.org/10.1016/j.neunet.2010.09.007

IF: 7.8

2010-01-01

Neural Networks

Abstract:This paper considers a class of online gradient learning methods for backpropagation (BP) neural networks with a single hidden layer. We assume that in each training cycle, each sample in the training set is supplied in a stochastic order to the network exactly once. It is interesting that these stochastic learning methods can be shown to be deterministically convergent. This paper presents some weak and strong convergence results for the learning methods, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. The conditions on the activation function and the learning rate to guarantee the convergence are relaxed compared with the existing results. Our convergence results are valid for not only S-S type neural networks (both the output and hidden neurons are Sigmoid functions), but also for P-P, P-S and S-P type neural networks, where S and P represent Sigmoid and polynomial functions, respectively.
Finite Convergence of MRI Neural Network for Linearly Separable Training Patterns

Lijun Liu,Wei Wu

DOI: https://doi.org/10.1007/978-3-540-28647-9_48

2004-01-01

Abstract:MRI (Madaline Rule I) neural network has wide applications. A finite convergence for the training of MRI neural network is proved for linearly separable training patterns.
A Nonlinear Adaptive Controller Based on RBF Networks

XH Chen,F Gao,JX Qian

DOI: https://doi.org/10.1109/icsmc.1996.569873

1996-01-01

Abstract:This paper proposes a nonlinear direct adaptive controller, based on radial basis function (RBF) networks. It is robust, reliable, efficient and simple. Compared with controllers based on BP networks, the proposed algorithm converges much more quickly without the problem of local minima. Simulation examples demonstrate the simplicity of the design procedure and the good characteristics of the control strategy. Moreover they illustrate that the controller possesses strong disturbance rejection and overcomes the drawback in outerpolation (accurate prediction outside the training domain) of neural network models.
Strong Convergence Of Gradient Methods For Bp Networks Training

Wei Wu,Hongmei Shao,Di Qu

DOI: https://doi.org/10.1109/icnnb.2005.1614626

2005-01-01

Abstract:Gradient method is a simple and popular learning algorithm for feedforward neural network (FNN) training. Some strong convergence results for both batch and online gradient methods are established based on existing weak convergence results. In particular, it is shown that for gradient-penalty algorithms, strong convergence results are immediate consequences of weak convergence results. For other batch and online gradient methods, the weak convergence plus the boundedness of the weights leads to the strong convergence. A class of gradient methods for general optimization problems is also considered, and some strong convergence results are obtained under mild conditions.
Convergence of BP Algorithm with Variable Learning Rates for FNN Training

Hongmei Shao,Wei Wu

DOI: https://doi.org/10.1109/MICAI.2006.11

2006-01-01

Abstract:Convergence results are presented for the batch backpropagation algorithm with variable learning rates for training feedforward neural networks with a hidden layer. The monotonicity of the error function in the training iteration is also proved.
A Theoretical View of Linear Backpropagation and Its Convergence

Ziang Li,Yiwen Guo,Haodi Liu,Changshui Zhang

DOI: https://doi.org/10.1109/tpami.2024.3353919

IF: 23.6

2024-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by Guo et al. [1]. Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al.'s paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments. Code for reproducing our experimental results will be publicly available.

computer science, artificial intelligence,engineering, electrical & electronic
Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Lu Xu,Jinshu Chen,Defeng Huang,Jianhua Lu,Licai Fang

DOI: https://doi.org/10.1109/tnnls.2013.2257845

IF: 14.255

2013-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved.
Global Convergence of Online Bp Training with Dynamic Learning Rate

Rui Zhang,Zong-Ben Xu,Guang-Bin Huang,Dianhui Wang

DOI: https://doi.org/10.1109/tnnls.2011.2178315

IF: 14.255

2012-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:The online backpropagation (BP) training procedure has been extensively explored in scientific research and engineering applications. One of the main factors affecting the performance of the online BP training is the learning rate. This paper proposes a new dynamic learning rate which is based on the estimate of the minimum error. The global convergence theory of the online BP training procedure with the proposed learning rate is further studied. It is proved that: 1) the error sequence converges to the global minimum error; and 2) the weight sequence converges to a fixed point at which the error function attains its global minimum. The obtained global convergence theory underlies the successful applications of the online BP training procedure. Illustrative examples are provided to support the theoretical analysis.
Convergence of Gradient Algorithm for BP Neural Network with Multiple Output Units

ZHOU Feng-qi,WU Wei,XIONG Yan

DOI: https://doi.org/10.3969/j.issn.1001-7011.2006.01.025

2006-01-01

Abstract:Gradient algorithm has been widely used for training the weights of feedforward neural networks. The convergence of the gradient algorithm for feedforward neural networks with one output unit has thoroughly studied. The convergence of gradient algorithm for a three-layer BP neural network with multiple output units is studied. The weak and strong convergence results of the corresponding gradient algorithm and the monotonicity of the error function in the iteration weighted vectors are proved.
Convergence of the PRP Based Online BP Training with Dynamic Step-Size Rule

张瑞,王伟,刘作志

DOI: https://doi.org/10.3969/j.issn.1672-4291.2012.02.003

2012-01-01

Abstract:The convergence of the PRP based online BP training with dynamic step-size rule（PRP-OLBP-DYN） is studied.It is shown that the sequence of the error functions converges to its global minimum error,and the sequence of the gradient of the error functions converges to zero.The experimental results verify the correctness of the obtained convergence results,and reveal the fast convergent speed and the good performance of the proposed PRP-OLBP-DYN algorithm.
Convergence of BP Algorithm for Product Unit Neural Networks with Exponential Weights.

C. Zhang,W. Wu,X. H. Chen,Y. Xiong

DOI: https://doi.org/10.1016/j.neucom.2007.12.004

IF: 6

2008-01-01

Neurocomputing

Abstract:Product unit neural networks with exponential weights (PUNNs) can provide more powerful internal representation capability than traditional feed-forward neural networks. In this paper, a convergence result of the back-propagation (BP) algorithm for training PUNNs is presented. The monotonicity of the error function in the training iteration process is also guaranteed. A numerical example is given to support the theoretical findings.
Convergence of batch BP algorithm with penalty for FNN training

Wei Wu,Hongmei Shao,Zhengxue Li

DOI: https://doi.org/10.1007/11893028_63

2006-01-01

Abstract:Penalty methods have been commonly used to improve the generalization performance of feedforward neural networks and to control the magnitude of the network weights. Weight boundedness and convergence results are presented for the batch BP algorithm with penalty for training feedforward neural networks with a hidden layer. A key point of the proofs is the monotonicity of the error function with the penalty term during the training iteration. A relationship between the learning rate parameter and the penalty parameter is proposed to guarantee the convergence. The algorithm is applied to two classification problems to support our theoretical findings.
Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks

Huisheng Zhang,Wei Wu

DOI: https://doi.org/10.1155/2009/329173

IF: 1.4

2009-01-01

Discrete Dynamics in Nature and Society

Abstract:The batch split-complex backpropagation (BSCBP) algorithm for training complex-valued neural networks is considered. For constant learning rate, it is proved that the error function of BSCBP algorithm is monotone during the training iteration process, and the gradient of the error function tends to zero. By adding a moderate condition, the weights sequence itself is also proved to be convergent. A numerical example is given to support the theoretical analysis.
Convergence of Online Gradient Method with a Penalty Term for BP Neural Network with Stochastic Inputs

LU Hui-fang,Wei Wu

2007-01-01

Abstract:In this paper,we present and discuss an online gradient method with a penalty term for three- layer BP neural networks.The input training examples are reset stochastically before the performance of each batch so that the learning is easy to jump off from local minima.The monotonicity and the convergence of deterministic nature are proved.
Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Qinwei Fan,Jacek M. Zurada,Wei Wu

DOI: https://doi.org/10.1016/j.neucom.2013.10.023

2014-01-01

Abstract:AbstractMinimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L"2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing L"1"/"2 regularization term. For normal L"1"/"2 regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal L"1"/"2 regularization term can be addressed. This paper shows the strong convergence results for the smoothing L"1"/"2 regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L"2 and normal L"1"/"2 regularizations respectively.
Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem

Tang, H.J.,Tan, K.C.,Zhang, Y.

DOI: https://doi.org/10.1109/IJCNN.2002.1007530

2002-01-01

Abstract:We study the convergence of a class of discrete recurrent neural networks to solve linear variational inequality problem (LVIP). LVIP has important applications in engineering and economics. Not only the network's exponential convergence for the case of positive definite matrix is proved, but its global convergence for positive semidefinite matrix is also proved. Conditions are derived to guarantee the convergences of the network. Comprehensive examples are discussed and simulated to illustrate the results

Finite Convergence of On-line BP Neural Networks with Linearly Separable Training Patterns

Convergence of online gradient methods for continuous perceptrons with linearly separable training patterns

A New Convergence Property of Online BP Learning

When does online BP training converge?

Convergence of BP Algorithm for Training MLP with Linear Output

Convergence Analysis of Online Gradient Method for BP Neural Networks

Finite Convergence of MRI Neural Network for Linearly Separable Training Patterns

A Nonlinear Adaptive Controller Based on RBF Networks

Strong Convergence Of Gradient Methods For Bp Networks Training

Convergence of BP Algorithm with Variable Learning Rates for FNN Training

A Theoretical View of Linear Backpropagation and Its Convergence

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Global Convergence of Online Bp Training with Dynamic Learning Rate

Convergence of Gradient Algorithm for BP Neural Network with Multiple Output Units

Convergence of the PRP Based Online BP Training with Dynamic Step-Size Rule

Convergence of BP Algorithm for Product Unit Neural Networks with Exponential Weights.

Convergence of batch BP algorithm with penalty for FNN training

Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks

Convergence of Online Gradient Method with a Penalty Term for BP Neural Network with Stochastic Inputs

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem