Convergence of On-Line Gradient Methods for Two-Layer Feedforward Neural Networks

Zheng-xue LI,Wei WU,Hong-wei ZHANG

DOI: https://doi.org/10.3770/j.issn.2095-2651.2001.02.011

2001-01-01

Abstract:A discussion is given on the convergence of the on-line gradient methods for two-layer feedforward neural networks in general cases. The theories are applied to some usual activation functions and energy functions.

What problem does this paper attempt to address?

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Lu Xu,Jinshu Chen,Defeng Huang,Jianhua Lu,Licai Fang

DOI: https://doi.org/10.1109/tnnls.2013.2257845

IF: 14.255

2013-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved.
Convergence of Gradient Method with Momentum for Two-Layer Feedforward Neural Networks

NM Zhang,W Wu,GF Zheng

DOI: https://doi.org/10.1109/tnn.2005.863460

2006-01-01

IEEE Transactions on Neural Networks

Abstract:A gradient method with momentum for two-layer feedforward neural networks is considered. The learning rate is set to be a constant and the momentum factor an adaptive variable. Both the weak and strong convergence results are proved, as well as the convergence rates for the error function and for the weight. Compared to the existing convergence results, our results are more general since we do not require the error function to be quadratic.
Convergence of an online gradient method with inner-product penalty and adaptive momentum

Hongmei Shao,Dongpo Xu,Gaofeng Zheng,Lijun Liu

DOI: https://doi.org/10.1016/j.neucom.2011.09.003

IF: 6

2012-01-01

Neurocomputing

Abstract:In this paper, we study the convergence of an online gradient method with inner-product penalty and adaptive momentum for feedforward neural networks, assuming that the training samples are permuted stochastically in each cycle of iteration. Both two-layer and three-layer neural network models are considered, and two convergence theorems are established. Sufficient conditions are proposed to prove weak and strong convergence results. The algorithm is applied to the classical two-spiral problem and identification of Gabor function problem to support these theoretical findings.
Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

Zhengxue Li,Wei Wu,Yulong Tian

DOI: https://doi.org/10.1016/j.cam.2003.08.062

IF: 2.872

2004-01-01

Journal of Computational and Applied Mathematics

Abstract:In this paper, we study the convergence of an online gradient method for feed-forward neural networks. The input training examples are permuted stochastically in each cycle of iteration. A monotonicity and a weak convergence of deterministic nature are proved.
Convergence of Online Gradient Method for Double Parallel Feedforward Neural Network

WANG Jian

DOI: https://doi.org/10.3969/j.issn.1672-6685.2012.01.005

2011-01-01

Abstract:The deterministic convergence for a Double Parallel Feedforward Neural Network (DPFNN) is studied. DPFNN is a parallel connection of a multi-layer feedforward neural network and a single layer feedforward neural network. Gradient method is used for training DPFNN with finite training sample set. The monotonicity of the error function in the training iteration is proved. Then, some weak and strong convergence results are obtained, indicating that the gradient of the error function tends to zero and the weight sequence goes to a fixed point, respectively. Numerical examples are provided, which support our theoretical findings and demonstrate that DPFNN has faster convergence speed and better generalization capability than the common feedforward neural network.
Convergence Analysis of Online Gradient Method for BP Neural Networks

Wei Wu,Jian Wang,Mingsong Cheng,Zhengxue Li

DOI: https://doi.org/10.1016/j.neunet.2010.09.007

IF: 7.8

2010-01-01

Neural Networks

Abstract:This paper considers a class of online gradient learning methods for backpropagation (BP) neural networks with a single hidden layer. We assume that in each training cycle, each sample in the training set is supplied in a stochastic order to the network exactly once. It is interesting that these stochastic learning methods can be shown to be deterministically convergent. This paper presents some weak and strong convergence results for the learning methods, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. The conditions on the activation function and the learning rate to guarantee the convergence are relaxed compared with the existing results. Our convergence results are valid for not only S-S type neural networks (both the output and hidden neurons are Sigmoid functions), but also for P-P, P-S and S-P type neural networks, where S and P represent Sigmoid and polynomial functions, respectively.
On the convergence of gradient descent for two layer neural networks

Lei Li

DOI: https://doi.org/10.48550/arXiv.1909.13671

2019-09-30

Optimization and Control

Abstract:It has been shown that gradient descent can yield the zero training loss in the over-parametrized regime (the width of the neural networks is much larger than the number of data points). In this work, combining the ideas of some existing works, we investigate the gradient descent method for training two-layer neural networks for approximating some target continuous functions. By making use the generic chaining technique from probability theory, we show that gradient descent can yield an exponential convergence rate, while the width of the neural networks needed is independent of the size of the training data. The result also implies some strong approximation ability of the two-layer neural networks without curse of dimensionality.
Convergence of gradient method for double parallel feedforward neural network

Jian Wang,Wei Wu,Zhengxue Li,Long Li

2011-01-01

International Journal of Numerical Analysis and Modeling

Abstract:The deterministic convergence for a Double Parallel Feedforward Neural Network (DPFNN) is studied. DPFNN is a parallel connection of a multi-layer feedforward neural network and a single layer feedforward neural network. Gradient method is used for training DPFNN with finite training sample set. The monotonicity of the error function in the training iteration is proved. Then, some weak and strong convergence results are obtained, indicating that the gradient of the error function tends to zero and the weight sequence goes to a fixed point, respectively. Numerical examples are provided, which support our theoretical findings and demonstrate that DPFNN has faster convergence speed and better generalization capability than the common feedforward neural network. © 2011 Institute for Scientific Computing and Information.
Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks

Hongmei Shao,Dongpo Xu,Gaofeng Zheng

DOI: https://doi.org/10.1007/s11063-011-9193-x

IF: 2.565

2011-01-01

Neural Processing Letters

Abstract:In this paper, a batch gradient algorithm with adaptive momentum is considered and a convergence theorem is presented when it is used for two-layer feedforward neural networks training. Simple but necessary sufficient conditions are offered to guarantee both weak and strong convergence. Compared with existing general requirements, we do not restrict the error function to be quadratic or uniformly convex. A numerical example is supplied to illustrate the performance of the algorithm and support our theoretical finding.
Convergence and monotonicity of an online gradient method with penalty for neural networks

Hongmei Shao,Wei Wu,Lijun Liu

2007-01-01

WSEAS TRANSACTIONS ON MATHEMATICS

Abstract:Penalty methods have been commonly used to improve the generalization of neural networks and to control the magnitude of network weights. Weight boundedness and convergence results are presented for the online gradient method with penalty for training a single-layer neural network. The monotonicity of the new error function during the training iteration is also proved. Finally, we apply the algorithm to a pattern classification problem to illustrate our theoretical findings.
Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Yuanzhi Li,Yang Yuan

DOI: https://doi.org/10.48550/arxiv.1705.09886

2017-01-01

Abstract:In recent years, stochastic gradient descent (SGD) based techniques has become the standard tools for training neural networks. However, formal theoretical understanding of why SGD can train neural networks in practice is largely missing. In this paper, we make progress on understanding this mystery by providing a convergence analysis for SGD on a rich subset of two-layer feedforward networks with ReLU activations. This subset is characterized by a special structure called "identity mapping". We prove that, if input follows from Gaussian distribution, with standard $O(1/\sqrt{d})$ initialization of the weights, SGD converges to the global minimum in polynomial number of steps. Unlike normal vanilla networks, the "identity mapping" makes our network asymmetric and thus the global minimum is unique. To complement our theory, we are also able to show experimentally that multi-layer networks with this mapping have better performance compared with normal vanilla networks. Our convergence theorem differs from traditional non-convex optimization techniques. We show that SGD converges to optimal in "two phases": In phase I, the gradient points to the wrong direction, however, a potential function $g$ gradually decreases. Then in phase II, SGD enters a nice one point convex region and converges. We also show that the identity mapping is necessary for convergence, as it moves the initial point to a better place for optimization. Experiment verifies our claims.
Convergence Analysis of Multilayer Feedforward Networks Trained with Penalty Terms: A review

Jian Wang,Guoling Yang,Shan Liu,Jacek M. Zurada

DOI: https://doi.org/10.1515/jacsm-2015-0011

2015-01-01

Abstract:Gradient descent method is one of the popular methods to train feedforward neural networks. Batch and incremental modes are the two most common methods to practically implement the gradient-based training for such networks. Furthermore, since generalization is an important property and quality criterion of a trained network, pruning algorithms with the addition of regularization terms have been widely used as an efficient way to achieve good generalization. In this paper, we review the convergence property and other performance aspects of recently researched training approaches based on different penalization terms. In addition, we show the smoothing approximation tricks when the penalty term is non-differentiable at origin.
Boundedness and Convergence of Online Gradient Method with Penalty and Momentum

Hongmei Shao,Gaofeng Zheng

DOI: https://doi.org/10.1016/j.neucom.2010.10.005

IF: 6

2011-01-01

Neurocomputing

Abstract:In this paper, the deterministic convergence of an online gradient method with penalty and momentum is investigated for training two-layer feedforward neural networks. The monotonicity of the new error function with the penalty term in the training iteration is firstly proved. Under this conclusion, we show that the weights are uniformly bounded during the training process and the algorithm is deterministically convergent. Sufficient conditions are also provided for both weak and strong convergence results.
Convergence of batch gradient algorithm for feedforward neural network training

Hongmei Shao,Wei Wu,Feng Li,Gaofeng Zheng

2007-01-01

Journal of Information and Computational Science

Abstract:Gradient algorithm has been widely used as a learning algorithm for training the weights of feedforward neural networks. A simple criterion is used to choose the learning rate during each cycle of training iteration. A convergence result of the corresponding batch gradient algorithm and the monotonicity error function in the iteration are proved, which are stronger than the usual results for general optimization problems. A supporting numerical experiment is also presented.
Convergence of A Gradient Algorithm with Penalty for Training Two-Layer Neural Networks

Hongmei Shao,Lijun Liu,Gaofeng Zheng

DOI: https://doi.org/10.1109/iccsit.2009.5234616

2009-01-01

Abstract:In this paper a squared penalty term is added to the conventional error function to improve the generalization of neural networks. A weight boundedness theorem and two convergence theorems are proved for the gradient learning algorithm with penalty when it is used for training a two-layer feedforward neural network. To illustrate above theoretical findings, numerical experiments are conducted based on a linearly separable problem and simulation results are presented. The abstract goes here.
Convergence of Gradient Algorithm for BP Neural Network with Multiple Output Units

ZHOU Feng-qi,WU Wei,XIONG Yan

DOI: https://doi.org/10.3969/j.issn.1001-7011.2006.01.025

2006-01-01

Abstract:Gradient algorithm has been widely used for training the weights of feedforward neural networks. The convergence of the gradient algorithm for feedforward neural networks with one output unit has thoroughly studied. The convergence of gradient algorithm for a three-layer BP neural network with multiple output units is studied. The weak and strong convergence results of the corresponding gradient algorithm and the monotonicity of the error function in the iteration weighted vectors are proved.
Boundedness and Convergence of Online Gradient Method with Penalty for Linear Output Feedforward Neural Networks

Huisheng Zhang,Wei Wu

DOI: https://doi.org/10.1007/s11063-009-9104-6

IF: 2.565

2009-01-01

Neural Processing Letters

Abstract:In this brief, we consider an online gradient method with penalty for training feedforward neural networks. Specifically, the penalty is a term proportional to the norm of the weights. Its roles in the method are to control the magnitude of the weights and to improve the generalization performance of the network. By proving that the weights are automatically bounded in the network training with penalty, we simplify the conditions that are required for convergence of online gradient method in literature. A numerical example is given to support the theoretical analysis.
Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Qinwei Fan,Jacek M. Zurada,Wei Wu

DOI: https://doi.org/10.1016/j.neucom.2013.10.023

2014-01-01

Abstract:AbstractMinimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L"2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing L"1"/"2 regularization term. For normal L"1"/"2 regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal L"1"/"2 regularization term can be addressed. This paper shows the strong convergence results for the smoothing L"1"/"2 regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L"2 and normal L"1"/"2 regularizations respectively.
CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

邵红梅,吴微,李峰

2005-01-01

Abstract:Online gradient algorithm has been widely used as a learning algorithm for feedforward neural network training. In this paper, we prove a weak convergence theorem of an online gradient algorithm with a penalty term, assuming that the training examples are input in a stochastic way. The monotonicity of the error function in the iteration and the boundedness of the weight are both guaranteed. We also present a numerical experiment to support our results.
Convergence Properties Analysis of Gradient Neural Network for Solving Online Linear Equations

Yu-Nong ZHANG,Zeng-Hai CHEN,Ke CHEN

DOI: https://doi.org/10.1016/s1874-1029(08)60104-4

2009-01-01

Abstract:A gradient neural network (GNN) for solving online a set of simultaneous linear equations is generalized and investigated in this paper. Instead of the earlier-presented asymptotical convergence, global exponential convergence could be proved for such a class of neural networks. In addition, superior convergence could be achieved using power-sigmoid activation-functions, compared with using linear activation-functions. Computer-simulation results substantiate further the above analysis and efficacy of such neural networks.

Convergence of On-Line Gradient Methods for Two-Layer Feedforward Neural Networks

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Convergence of Gradient Method with Momentum for Two-Layer Feedforward Neural Networks

Convergence of an online gradient method with inner-product penalty and adaptive momentum

Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

Convergence of Online Gradient Method for Double Parallel Feedforward Neural Network

Convergence Analysis of Online Gradient Method for BP Neural Networks

On the convergence of gradient descent for two layer neural networks

Convergence of gradient method for double parallel feedforward neural network

Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks

Convergence and monotonicity of an online gradient method with penalty for neural networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Convergence Analysis of Multilayer Feedforward Networks Trained with Penalty Terms: A review

Boundedness and Convergence of Online Gradient Method with Penalty and Momentum

Convergence of batch gradient algorithm for feedforward neural network training

Convergence of A Gradient Algorithm with Penalty for Training Two-Layer Neural Networks

Convergence of Gradient Algorithm for BP Neural Network with Multiple Output Units

Boundedness and Convergence of Online Gradient Method with Penalty for Linear Output Feedforward Neural Networks

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Convergence Properties Analysis of Gradient Neural Network for Solving Online Linear Equations