Convergence of online gradient methods for continuous perceptrons with linearly separable training patterns

Wei Wu,Zhiqiong Shao

DOI: https://doi.org/10.1016/S0893-9659(03)90086-3

IF: 4.294

2003-01-01

Applied Mathematics Letters

Abstract:In this paper, we prove that the online gradient method for continuous perceptrons converges in finite steps when the training patterns are linearly separable.

What problem does this paper attempt to address?

Finite Convergence of On-line BP Neural Networks with Linearly Separable Training Patterns

Zhi-qiong SHAO,Wei WU,Jie YANG

DOI: https://doi.org/10.3770/j.issn.2095-2651.2006.03.004

2006-01-01

Abstract:In this paper we prove a finite convergence of online BP algorithms for nonlinear feedforward neural networks when the training patterns are linearly separable.
The convergence of conjugate gradient method with nonmonotone line search

Zhen-Jun Shi,Shengquan Wang,Zhiwei Xu

DOI: https://doi.org/10.1016/j.amc.2010.06.047

IF: 4.397

2010-01-01

Applied Mathematics and Computation

Abstract:The conjugate gradient method is a useful and powerful approach for solving large-scale minimization problems. Liu and Storey developed a conjugate gradient method, which has good numerical performance but no global convergence under traditional line searches such as Armijo line search, Wolfe line search, and Goldstein line search. In this paper we propose a new nonmonotone line search for Liu-Storey conjugate gradient method (LS in short). The new nonmonotone line search can guarantee the global convergence of LS method and has a good numerical performance. By estimating the Lipschitz constant of the derivative of objective functions in the new nonmonotone line search, we can find an adequate step size and substantially decrease the number of functional evaluations at each iteration. Numerical results show that the new approach is effective in practical computation.
A new training algorithm for a fuzzy perceptron and its convergence

Jie Yang,Wei Wu,Zhiqiong Shao

DOI: https://doi.org/10.1007/11427391_97

2005-01-01

Abstract:In this paper, we present a new training algorithm for a fuzzy perceptron. In the case where the dimension of the input vectors is two and the training examples are separable, we can prove a finite convergence, i.e., the training procedure for the network weights will stop after finite steps. When the dimension is greater than two, stronger conditions are needed to guarantee the finite convergence.
Convergence Analysis of Online Gradient Method for BP Neural Networks

Wei Wu,Jian Wang,Mingsong Cheng,Zhengxue Li

DOI: https://doi.org/10.1016/j.neunet.2010.09.007

IF: 7.8

2010-01-01

Neural Networks

Abstract:This paper considers a class of online gradient learning methods for backpropagation (BP) neural networks with a single hidden layer. We assume that in each training cycle, each sample in the training set is supplied in a stochastic order to the network exactly once. It is interesting that these stochastic learning methods can be shown to be deterministically convergent. This paper presents some weak and strong convergence results for the learning methods, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. The conditions on the activation function and the learning rate to guarantee the convergence are relaxed compared with the existing results. Our convergence results are valid for not only S-S type neural networks (both the output and hidden neurons are Sigmoid functions), but also for P-P, P-S and S-P type neural networks, where S and P represent Sigmoid and polynomial functions, respectively.
Convergence of Gradient Method for Elman Networks

Wei Wu,Dong-po Xu,Zheng-xue Li

DOI: https://doi.org/10.1007/s10483-008-0912-z

2008-01-01

Applied Mathematics and Mechanics

Abstract:The gradient method for training Elman networks with a finite training sample set is considered. Monotonicity of the error function in the iteration is shown. Weak and strong convergence results are proved, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. A numerical example is given to support the theoretical findings.
Convergence and monotonicity of an online gradient method with penalty for neural networks

Hongmei Shao,Wei Wu,Lijun Liu

2007-01-01

WSEAS TRANSACTIONS ON MATHEMATICS

Abstract:Penalty methods have been commonly used to improve the generalization of neural networks and to control the magnitude of network weights. Weight boundedness and convergence results are presented for the online gradient method with penalty for training a single-layer neural network. The monotonicity of the new error function during the training iteration is also proved. Finally, we apply the algorithm to a pattern classification problem to illustrate our theoretical findings.
Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Lu Xu,Jinshu Chen,Defeng Huang,Jianhua Lu,Licai Fang

DOI: https://doi.org/10.1109/tnnls.2013.2257845

IF: 14.255

2013-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved.
Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Qinwei Fan,Jacek M. Zurada,Wei Wu

DOI: https://doi.org/10.1016/j.neucom.2013.10.023

2014-01-01

Abstract:AbstractMinimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L"2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing L"1"/"2 regularization term. For normal L"1"/"2 regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal L"1"/"2 regularization term can be addressed. This paper shows the strong convergence results for the smoothing L"1"/"2 regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L"2 and normal L"1"/"2 regularizations respectively.
A Discussion of the Learning in a Simple Perceptron

Jianfeng Feng

1995-01-01

Abstract:In this paper,we extend the convergence of the simple perceptron learning rule to the case that the set of inputs is infinity or a region。 When the set of inputs is linearly separable,we prove that a simple perceptron always improves its performance。As the set of the inputs is ‘strong’linearly separable,then within finite time the connections among units converge to a limit which separates the inputs.The convergence rate is also estimated。
Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

Zhengxue Li,Wei Wu,Yulong Tian

DOI: https://doi.org/10.1016/j.cam.2003.08.062

IF: 2.872

2004-01-01

Journal of Computational and Applied Mathematics

Abstract:In this paper, we study the convergence of an online gradient method for feed-forward neural networks. The input training examples are permuted stochastically in each cycle of iteration. A monotonicity and a weak convergence of deterministic nature are proved.
Strong Convergence Of Gradient Methods For Bp Networks Training

Wei Wu,Hongmei Shao,Di Qu

DOI: https://doi.org/10.1109/icnnb.2005.1614626

2005-01-01

Abstract:Gradient method is a simple and popular learning algorithm for feedforward neural network (FNN) training. Some strong convergence results for both batch and online gradient methods are established based on existing weak convergence results. In particular, it is shown that for gradient-penalty algorithms, strong convergence results are immediate consequences of weak convergence results. For other batch and online gradient methods, the weak convergence plus the boundedness of the weights leads to the strong convergence. A class of gradient methods for general optimization problems is also considered, and some strong convergence results are obtained under mild conditions.
Convergence of gradient method for double parallel feedforward neural network

Jian Wang,Wei Wu,Zhengxue Li,Long Li

2011-01-01

International Journal of Numerical Analysis and Modeling

Abstract:The deterministic convergence for a Double Parallel Feedforward Neural Network (DPFNN) is studied. DPFNN is a parallel connection of a multi-layer feedforward neural network and a single layer feedforward neural network. Gradient method is used for training DPFNN with finite training sample set. The monotonicity of the error function in the training iteration is proved. Then, some weak and strong convergence results are obtained, indicating that the gradient of the error function tends to zero and the weight sequence goes to a fixed point, respectively. Numerical examples are provided, which support our theoretical findings and demonstrate that DPFNN has faster convergence speed and better generalization capability than the common feedforward neural network. © 2011 Institute for Scientific Computing and Information.
Convergence of On-Line Gradient Methods for Two-Layer Feedforward Neural Networks

Zheng-xue LI,Wei WU,Hong-wei ZHANG

DOI: https://doi.org/10.3770/j.issn.2095-2651.2001.02.011

2001-01-01

Abstract:A discussion is given on the convergence of the on-line gradient methods for two-layer feedforward neural networks in general cases. The theories are applied to some usual activation functions and energy functions.
When does online BP training converge?

Zong-Ben Xu,Rui Zhang,Wen-Feng Jing

DOI: https://doi.org/10.1109/TNN.2009.2025946

2009-01-01

Abstract:The backpropogation (BP) neural networks have been widely applied in scientific research and engineering. The success of the application, however, relies upon the convergence of the training procedure involved in the neural network learning. We settle down the convergence analysis issue through proving two fundamental theorems on the convergence of the online BP training procedure. One theorem claims that under mild conditions, the gradient sequence of the error function will converge to zero (the weak convergence), and another theorem concludes the convergence of the weight sequence defined by the procedure to a fixed value at which the error function attains its minimum (the strong convergence). The weak convergence theorem sharpens and generalizes the existing convergence analysis conducted before, while the strong convergence theorem provides new analysis results on convergence of the online BP training procedure. The results obtained reveal that with any analytic sigmoid activation function, the online BP training procedure is always convergent, which then underlies successful application of the BP neural networks.
Convergence of Gradient Method for A Fully Recurrent Neural Network

Dongpo Xu,Zhengxue Li,Wei Wu

DOI: https://doi.org/10.1007/s00500-009-0398-0

IF: 3.732

2009-01-01

Soft Computing

Abstract:Recurrent neural networks have been successfully used for analysis and prediction of temporal sequences. This paper is concerned with the convergence of a gradient-descent learning algorithm for training a fully recurrent neural network. In literature, stochastic process theory has been used to establish some convergence results of probability nature for the on-line gradient training algorithm, based on the assumption that a very large number of (or infinitely many in theory) training samples of the temporal sequences are available. In this paper, we consider the case that only a limited number of training samples of the temporal sequences are available such that the stochastic treatment of the problem is no longer appropriate. Instead, we use an off-line gradient training algorithm for the fully recurrent neural network, and we accordingly prove some convergence results of deterministic nature. The monotonicity of the error function in the iteration is also guaranteed. A numerical example is given to support the theoretical findings.
On Convergence of Training Loss Without Reaching Stationary Points

Jingzhao Zhang,Haochuan Li,Suvrit Sra,Ali Jadbabaie

2021-01-01

Abstract: It is a well-known fact that nonconvex optimization is computationally intractable in the worst case. As a result, theoretical analysis of optimization algorithms such as gradient descent often focuses on local convergence to stationary points where the gradient norm is zero or negligible. In this work, we examine the disconnect between the existing theoretical analysis of gradient-based algorithms and actual practice. Specifically, we provide numerical evidence that in large-scale neural network training, such as in ImageNet, ResNet, and WT103 + TransformerXL models, the Neural Network weight variables do not converge to stationary points where the gradient of the loss function vanishes. Remarkably, however, we observe that while weights do not converge to stationary points, the value of the loss function converges. Inspired by this observation, we propose a new perspective based on ergodic theory of dynamical systems. We prove convergence of the distribution of weight values to an approximate invariant measure (without smoothness assumptions) that explains this phenomenon. We further discuss how this perspective can better align the theory with empirical observations.
CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

邵红梅,吴微,李峰

2005-01-01

Abstract:Online gradient algorithm has been widely used as a learning algorithm for feedforward neural network training. In this paper, we prove a weak convergence theorem of an online gradient algorithm with a penalty term, assuming that the training examples are input in a stochastic way. The monotonicity of the error function in the iteration and the boundedness of the weight are both guaranteed. We also present a numerical experiment to support our results.
Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

Xiangmei Wang,Chong Li,Jinhua Wang,Jen-Chih Yao

DOI: https://doi.org/10.1137/14099961x

IF: 2.763

2015-01-01

SIAM Journal on Optimization

Abstract:We study the convergence issue of the subgradient algorithm for solving the convex feasibility problems in Riemannian manifolds, which was first proposed and analyzed by Bento and Melo [J. Optim. Theory Appl., 152 (2012), pp. 773-785]. The linear convergence property about the subgradient algorithm for solving the convex feasibility problems with the Slater condition in Riemannian manifolds are established, and some step sizes rules are suggested for finite convergence purposes, which are motivated by the work due to De Pierro Iusem [Appl. Math. Optim., 17 (1988), pp. 225-235]. As a by-product, the convergence result of this algorithm is obtained for the convex feasibility problem without the Slater condition assumption. These results extend and/or improve the corresponding known ones in both the Euclidean space and Riemannian manifolds.
A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Sanjeev Arora,Nadav Cohen,Noah Golowich,Wei Hu

DOI: https://doi.org/10.48550/arXiv.1810.02281

IF: 5.414

2018-10-04

Machine Learning

Abstract:We analyze speed of convergence to global optimum for gradient descent training a deep linear neural network (parameterized as $x \mapsto W_N W_{N-1} \cdots W_1 x$) by minimizing the $\ell_2$ loss over whitened data. Convergence at a linear rate is guaranteed when the following hold: (i) dimensions of hidden layers are at least the minimum of the input and output dimensions; (ii) weight matrices at initialization are approximately balanced; and (iii) the initial loss is smaller than the loss of any rank-deficient solution. The assumptions on initialization (conditions (ii) and (iii)) are necessary, in the sense that violating any one of them may lead to convergence failure. Moreover, in the important case of output dimension 1, i.e. scalar regression, they are met, and thus convergence to global optimum holds, with constant probability under a random initialization scheme. Our results significantly extend previous analyses, e.g., of deep linear residual networks (Bartlett et al., 2018).
On Accelerated Perceptrons and Beyond

Guanghui Wang,Rafael Hanashiro,Etash Guha,Jacob Abernethy

DOI: https://doi.org/10.48550/arXiv.2210.09371

2022-10-18

Abstract:The classical Perceptron algorithm of Rosenblatt can be used to find a linear threshold function to correctly classify $n$ linearly separable data points, assuming the classes are separated by some margin $\gamma > 0$. A foundational result is that Perceptron converges after $\Omega(1/\gamma^{2})$ iterations. There have been several recent works that managed to improve this rate by a quadratic factor, to $\Omega(\sqrt{\log n}/\gamma)$, with more sophisticated algorithms. In this paper, we unify these existing results under one framework by showing that they can all be described through the lens of solving min-max problems using modern acceleration techniques, mainly through optimistic online learning. We then show that the proposed framework also lead to improved results for a series of problems beyond the standard Perceptron setting. Specifically, a) For the margin maximization problem, we improve the state-of-the-art result from $O(\log t/t^2)$ to $O(1/t^2)$, where $t$ is the number of iterations; b) We provide the first result on identifying the implicit bias property of the classical Nesterov's accelerated gradient descent (NAG) algorithm, and show NAG can maximize the margin with an $O(1/t^2)$ rate; c) For the classical $p$-norm Perceptron problem, we provide an algorithm with $\Omega(\sqrt{(p-1)\log n}/\gamma)$ convergence rate, while existing algorithms suffer the $\Omega({(p-1)}/\gamma^2)$ convergence rate.

Machine Learning,Optimization and Control

Convergence of online gradient methods for continuous perceptrons with linearly separable training patterns

Finite Convergence of On-line BP Neural Networks with Linearly Separable Training Patterns

The convergence of conjugate gradient method with nonmonotone line search

A new training algorithm for a fuzzy perceptron and its convergence

Convergence Analysis of Online Gradient Method for BP Neural Networks

Convergence of Gradient Method for Elman Networks

Convergence and monotonicity of an online gradient method with penalty for neural networks

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

A Discussion of the Learning in a Simple Perceptron

Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

Strong Convergence Of Gradient Methods For Bp Networks Training

Convergence of gradient method for double parallel feedforward neural network

Convergence of On-Line Gradient Methods for Two-Layer Feedforward Neural Networks

When does online BP training converge?

Convergence of Gradient Method for A Fully Recurrent Neural Network

On Convergence of Training Loss Without Reaching Stationary Points

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

On Accelerated Perceptrons and Beyond