Convergence of gradient descent algorithm for Pi-sigma neural networks

Huifang Lu,Wei Wu,Chao Zhang,Yan Xiong

2006-01-01

Journal of Information and Computational Science

Abstract:Pi-sigma neural networks (PSNN) are a class of feed-forward polynomial neural networks. This paper is concerned with the gradient descent algorithm for the network training of PSNN. Monotonicity and convergence of this method are proved.

What problem does this paper attempt to address?

Convergence of online gradient method for pi-sigma neural networks

Yan Xiong,Wei Wu,Huifang Lu,Chao Zhang

2007-01-01

Journal of Computational Information Systems

Abstract:Pi-sigma neural networks are a class of feed-forward polynomial networks, and have been used effectively in classification and approximation problems. They are also used as basic building blocks for other more complicated high-order networks because of their powerful nonlinear descriptive capacity with fewer number of weights. An online gradient learning method for pi-sigma neural networks is considered in this paper. The monotonicity of the error function in the iterative learning process is guaranteed. And a weak convergence result of the method is proved, indicating that the gradient of the error function goes to zero. A strong convergence result is also obtained under the condition that the weights keep bounded in the learning process, indicating that the weight sequence converges to a local minimum of the error function. The learning algorithm is applied to a four-dimensional parity problem to support our theoretical findings.
Convergence of online gradient methods for Pi-Sigma neural network with a penalty term

Yan Liu,Huisheng Zhang,Jie Yang,Wu, Wei

DOI: https://doi.org/10.1109/ANTHOLOGY.2013.6784769

2013-01-01

Abstract:Pi-Sigma network is a class of feed-forward neural networks with product units in the output layer. An online gradient method with penalty which is proportional to the norm of the weights is presented and the convergence results are discussed for a Pi-Sigma neural network. The weights are proved to be bounded in the training process, thus the conditions that are required for convergence of online gradient method in literature are simplified.
Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Xidai Kang,Yan Xiong,Chao Zhang,Wei Wu

DOI: https://doi.org/10.1109/foci.2007.371528

2007-01-01

Abstract:An online gradient method is presented and discussed for Pi-Sigma neural networks with stochastic inputs. The error function is proved to be monotone in the training process, and the gradient of the error function tends to zero if the weights sequence is uniformly bounded. Furthermore, after adding a moderate condition, the weights sequence itself is also proved to be convergent
Convergence Analysis of Batch Gradient Algorithm for Three Classes of Sigma-Pi Neural Networks

Chao Zhang,Wei Wu,Yan Xiong

DOI: https://doi.org/10.1007/s11063-007-9050-0

IF: 2.565

2007-01-01

Neural Processing Letters

Abstract:Sigma-Pi (Σ-Π) neural networks (SPNNs) are known to provide more powerful mapping capability than traditional feed-forward neural networks. A unified convergence analysis for the batch gradient algorithm for SPNN learning is presented, covering three classes of SPNNs: Σ-Π-Σ, Σ-Σ-Π and Σ-Π-Σ-Π. The monotonicity of the error function in the iteration is also guaranteed.
Training Pi-Sigma Network by Online Gradient Algorithm with Penalty for Small Weight Update.

Yan Xiong,Wei Wu,Xidai Kang,Chao Zhang

DOI: https://doi.org/10.1162/neco.2007.19.12.3356

IF: 3.278

2007-01-01

Neural Computation

Abstract:A pi-sigma network is a class of feedforward neural networks with product units in the output layer. An online gradient algorithm is the simplest and most often used training method for feedforward neural networks. But there arises a problem when the online gradient algorithm is used for pi-sigma networks in that the update increment of the weights may become very small, especially early in training, resulting in a very slow convergence. To overcome this difficulty, we introduce an adaptive penalty term into the error function, so as to increase the magnitude of the update increment of the weights when it is too small. This strategy brings about faster convergence as shown by the numerical experiments carried out in this letter.
Convergence of Batch Gradient Learning Algorithm with Smoothing L1/2 Regularization for Sigma–Pi–Sigma Neural Networks

Yan Liu,Zhengxue Li,Dakun Yang,Kh. Sh. Mohamed,Jing Wang,Wei Wu

DOI: https://doi.org/10.1016/j.neucom.2014.09.031

IF: 6

2015-01-01

Neurocomputing

Abstract:Sigma-Pi-Sigma neural networks are known to provide more powerful mapping capability than traditional feed-forward neural networks. The L-1/2 regularizer is very useful and efficient, and can be taken as a representative of all the L-q(0 < q < 1) regularizers. However, the nonsmoothness of L-1/2 regulaiization may lead to oscillation phenomenon. The aim of this paper is to develop a novel batch gradient method with smoothing L-1/2 regularization for Sigma-Pi-Sigma neural networks. Compared with conventional gradient learning algorithm, this method produces sparser weights and simpler structure, and it improves the learning efficiency. A comprehensive study on the weak and strong convergence results for this algorithm are also presented, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed value, respectively. (C) 2014 Elsevier B.V. All rights reserved.
A Modified Higher-Order Feed Forward Neural Network with Smoothing Regularization

Kh. Sh. Mohamed,W. Wu,Y. Liu

DOI: https://doi.org/10.14311/nnw.2017.27.032

2017-01-01

Neural Network World

Abstract:This paper proposes an offline gradient method with smoothing L1/2 regularization for learning and pruning of the pi-sigma neural networks (PSNNs). The original L1/2 regularization term is not smooth at the origin, since it involves the absolute value function. This causes oscillation in the computation and difficulty in the convergence analysis. In this paper, we propose to use a smooth function to replace and approximate the absolute value function, ending up with a smoothing L1/2 regularization method for PSNN. Numerical simulations show that the smoothing L1/2 regularization method eliminates the oscillation in computation and achieves better learning accuracy. We are also able to prove a convergence theorem for the proposed learning method.
A Modifled Gradient-Based Neuro-Fuzzy Learning Algorithm for Pi-Sigma Network Based on First-Order Takagi-Sugeno System

Yan Liu,Jie Yang,Dakun Yang,Wei Wu

2014-01-01

Abstract:This paper presents a Pi-Sigma network to identify flrst-order Tagaki-Sugeno (T-S) fuzzy inference system and proposes a simplifled gradient-based neuro-fuzzy learning algorithm. A comprehensive study on the weak and strong convergence for the learning method is made, which indicates that the sequence of error function goes to a flxed value, and the gradient of the error function goes to zero, respectively.

Computer Science
Convergence of gradient method for double parallel feedforward neural network

Jian Wang,Wei Wu,Zhengxue Li,Long Li

2011-01-01

International Journal of Numerical Analysis and Modeling

Abstract:The deterministic convergence for a Double Parallel Feedforward Neural Network (DPFNN) is studied. DPFNN is a parallel connection of a multi-layer feedforward neural network and a single layer feedforward neural network. Gradient method is used for training DPFNN with finite training sample set. The monotonicity of the error function in the training iteration is proved. Then, some weak and strong convergence results are obtained, indicating that the gradient of the error function tends to zero and the weight sequence goes to a fixed point, respectively. Numerical examples are provided, which support our theoretical findings and demonstrate that DPFNN has faster convergence speed and better generalization capability than the common feedforward neural network. © 2011 Institute for Scientific Computing and Information.
A new Sigma-Pi-Sigma neural network based on $ L_1 $ and $ L_2 $ regularization and applications

Jianwei Jiao,Keqin Su

DOI: https://doi.org/10.3934/math.2024293

2024-01-01

AIMS Mathematics

Abstract:As one type of the important higher-order neural networks developed in the last decade, the Sigma-Pi-Sigma neural network has more powerful nonlinear mapping capabilities compared with other popular neural networks. This paper is concerned with a new Sigma-Pi-Sigma neural network based on a $ L_1 $ and $ L_2 $ regularization batch gradient method, and the numerical experiments for classification and regression problems prove that the proposed algorithm is effective and has better properties comparing with other classical penalization methods. The proposed model combines the sparse solution tendency of $ L_1 $ norm and the high benefits in efficiency of the $ L_2 $ norm, which can regulate the complexity of a network and prevent overfitting. Also, the numerical oscillation, induced by the non-differentiability of $ L_1 $ plus $ L_2 $ regularization at the origin, can be eliminated by a smoothing technique to approximate the objective function.

mathematics, applied
Convergence of Stochastic Gradient Descent in Deep Neural Network

Bai-cun Zhou,Cong-ying Han,Tian-de Guo

DOI: https://doi.org/10.1007/s10255-021-0991-2

2021-01-01

Abstract:Stochastic gradient descent (SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning. This algorithm and its variants are the preferred algorithm while optimizing parameters of deep neural network for their advantages of low storage space requirement and fast computation speed. Previous studies on convergence of these algorithms were based on some traditional assumptions in optimization problems. However, the deep neural network has its unique properties. Some assumptions are inappropriate in the actual optimization process of this kind of model. In this paper, we modify the assumptions to make them more consistent with the actual optimization process of deep neural network. Based on new assumptions, we studied the convergence and convergence rate of SGD and its two common variant algorithms. In addition, we carried out numerical experiments with LeNet-5, a common network framework, on the data set MNIST to verify the rationality of our assumptions.
An improved asynchronous batch gradient method for ridge polynomial neural network

Yan Xiong,Shumei He

DOI: https://doi.org/10.1016/j.neucom.2024.127529

IF: 6

2024-03-08

Neurocomputing

Abstract:The ridge polynomial neural network composed of pi-sigma modules is a typical higher-order feedforward network, which has good non-linear mapping capabilities. Due to the strong coupling of the network structure, the synchronous gradient method can easily result in significant fluctuations in updated weights. This will reduce the generalization ability of the ridge polynomial neural network for solving classification problems. In this paper, the proposed method is a novel improved asynchronous batch gradient method, and combined with the adaptive parameters of the activation function are trained synchronously. We strictly prove the convergence theorem of the improved method. The numerical experimental results also indicate that our method for training ridge polynomial neural networks can help the weight changes smoothly and the network has good generalization ability. The feasibility and effectiveness of our method are verified from both theoretical and experimental perspectives.

computer science, artificial intelligence
Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks

Xianliang Xu,Ting Du,Wang Kong,Ye Li,Zhongyi Huang

2024-08-06

Abstract:First-order methods, such as gradient descent (GD) and stochastic gradient descent (SGD), have been proven effective in training neural networks. In the context of over-parameterization, there is a line of work demonstrating that randomly initialized (stochastic) gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. However, the learning rate of GD for training two-layer neural networks exhibits poor dependence on the sample size and the Gram matrix, leading to a slow training process. In this paper, we show that for the $L^2$ regression problems, the learning rate can be improved from $\mathcal{O}(\lambda_0/n^2)$ to $\mathcal{O}(1/\|\bm{H}^{\infty}\|_2)$, which implies that GD actually enjoys a faster convergence rate. Furthermore, we generalize the method to GD in training two-layer Physics-Informed Neural Networks (PINNs), showing a similar improvement for the learning rate. Although the improved learning rate has a mild dependence on the Gram matrix, we still need to set it small enough in practice due to the unknown eigenvalues of the Gram matrix. More importantly, the convergence rate is tied to the least eigenvalue of the Gram matrix, which can lead to slow convergence. In this work, we provide the convergence analysis of natural gradient descent (NGD) in training two-layer PINNs, demonstrating that the learning rate can be $\mathcal{O}(1)$, and at this rate, the convergence rate is independent of the Gram matrix.

Machine Learning
Convergence and monotonicity of an online gradient method with penalty for neural networks

Hongmei Shao,Wei Wu,Lijun Liu

2007-01-01

WSEAS TRANSACTIONS ON MATHEMATICS

Abstract:Penalty methods have been commonly used to improve the generalization of neural networks and to control the magnitude of network weights. Weight boundedness and convergence results are presented for the online gradient method with penalty for training a single-layer neural network. The monotonicity of the new error function during the training iteration is also proved. Finally, we apply the algorithm to a pattern classification problem to illustrate our theoretical findings.
On the Convergence and Improvement of Stochastic Normalized Gradient Descent

Zhao Shen-Yi,Xie Yin-Peng,Li Wu-Jun

DOI: https://doi.org/10.1007/s11432-020-3023-7

2021-01-01

Science China Information Sciences

Abstract:Non-convex models, like deep neural networks, have been widely used in machine learning applications. Training non-convex models is a difficult task owing to the saddle points of models. Recently,stochastic normalized gradient descent(SNGD), which updates the model parameter by a normalized gradient in each iteration, has attracted much attention. Existing results show that SNGD can achieve better performance on escaping saddle points than classical training methods like stochastic gradient descent(SGD).However, none of the existing studies has provided theoretical proof about the convergence of SNGD for non-convex problems. In this paper, we firstly prove the convergence of SNGD for non-convex problems.Particularly, we prove that SNGD can achieve the same computation complexity as SGD. In addition, based on our convergence proof of SNGD, we find that SNGD needs to adopt a small constant learning rate for convergence guarantee. This makes SNGD do not perform well on training large non-convex models in practice. Hence, we propose a new method, called stagewise SNGD(S-SNGD), to improve the performance of SNGD. Different from SNGD in which a small constant learning rate is necessary for convergence guarantee,S-SNGD can adopt a large initial learning rate and reduce the learning rate by stage. The convergence of S-SNGD can also be theoretically proved for non-convex problems. Empirical results on deep neural networks show that S-SNGD achieves better performance than SNGD in terms of both training loss and test accuracy.
On Faster Convergence of Scaled Sign Gradient Descent

Xiuxian Li,Kuo-Yi Lin,Li Li,Yiguang Hong,Jie Chen

DOI: https://doi.org/10.1109/tii.2023.3280938

IF: 12.3

2023-01-01

IEEE Transactions on Industrial Informatics

Abstract:Communication has been seen as a significant bottleneck in industrial applications over large-scale networks. To alleviate the communication burden, sign-based optimization algorithms have gained popularity recently in both industrial and academic communities, which is shown to be closely related to adaptive gradient methods, such as Adam. Along this line, this paper investigates faster convergence for a variant of sign-based gradient descent, called scaled SIGNGD, in three cases: 1) the objective function is strongly convex; 2) the objective function is nonconvex but satisfies the Polyak-Łojasiewicz (PL) inequality; 3) the gradient is stochastic, called scaled SIGNSGD in this case. For the first two cases, it can be shown that the scaled SIGNGD converges at a linear rate. For case 3), the algorithm is shown to converge linearly to a neighborhood of the optimal value when a constant learning rate is employed, and the algorithm converges at a rate of $O(1/k+1/k^{2}+1/k^{3})$ when using a diminishing learning rate, where $k$ is the iteration number. The results are also extended to the distributed setting by majority vote in a parameter-server framework. Finally, numerical experiments are performed to corroborate the theoretical findings.

automation & control systems,computer science, interdisciplinary applications,engineering, industrial
A Polak-Ribière-Polyak Conjugate Gradient-Based Neuro-Fuzzy Network and Its Convergence

Tao Gao,Jian Wang,Bingjie Zhang,Huaqing Zhang,Peng Ren,Nikhil R. Pal

DOI: https://doi.org/10.1109/access.2018.2848117

IF: 3.9

2018-01-01

IEEE Access

Abstract:Conjugate gradient methods have advantages, such as fast convergence and low memory requirement, which are important for many real-life applications. For zero-order Takagi-Sugeno inference systems, a Polak-Ribiere-Polyak conjugate gradient-based algorithm is proposed to train a neuro-fuzzy network. Compared with the existing gradient-based training algorithm, this scheme efficiently enhances the learning performance. Two deterministic convergence results are proved in detail, which indicate that the gradient of the objective function approaches zero (weak convergence) and the sequence of system parameters tends to a fixed point (strong convergence), respectively. To demonstrate the effectiveness of the proposed algorithm, as well as, to validate the theoretical results, numerical simulations are provided for both function approximation and classification type systems.
Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Jian Wang,Bingjie Zhang,Zhaoyang Sang,Yusong Liu,Shujun Wu,Quan Miao

DOI: https://doi.org/10.1007/s00521-018-3748-y

2018-09-29

Neural Computing and Applications

Abstract:Based on a novel algorithm, known as the upper-layer-solution-aware (USA), a new algorithm, in which the penalty method is introduced into the empirical risk, is studied for training feed-forward neural networks in this paper, named as USA with penalty. Both theoretical analysis and numerical results show that it can control the magnitude of weights of the networks. Moreover, the deterministic theoretical analysis of the new algorithm is proved. The monotonicity of the empirical risk with penalty term is guaranteed in the training procedure. The weak and strong convergence results indicate that the gradient of the total error function with respect to weights tends to zero, and the weight sequence goes to a fixed point when the iterations approach positive infinity. Numerical experiment has been implemented and effectively verifies the proved theoretical results.

computer science, artificial intelligence
On the Convergence of Decentralized Gradient Descent.

Kun Yuan,Qing Ling,Wotao Yin

DOI: https://doi.org/10.1137/130943170

IF: 2.763

2016-01-01

SIAM Journal on Optimization

Abstract:Consider the consensus problem of minimizing f(x) = Sigma(n)(i=1) fi(x), where x is an element of R-p and each f(i) is only known to the individual agent i in a connected network of n agents. To solve this problem and obtain the solution, all the agents collaborate with their neighbors through information exchange. This type of decentralized computation does not need a fusion center, offers better network load balance, and improves data privacy. This paper studies the decentralized gradient descent method [A. Nedic and A. Ozdaglar, IEEE Trans. Automat. Control, 54 (2009), pp. 48-61], in which each agent i updates its local variable x((i)) is an element of R-n by combining the average of its neighbors' with a local negative-gradient step -alpha del f(i)(x((i))). The method is described by the iteration x((i)) (k + 1) <- Sigma(n)(j=1) w(ij)x((j))(k)-alpha del f(i)(x((i))(k)), for each agent i, where w(ij) is nonzero only if i and j are neighbors or i = j and the matrix W = [w(ij)] is an element of R-n (x) (n) is symmetric and doubly stochastic. This paper analyzes the convergence of this iteration and derives its rate of convergence under the assumption that each f(i) is proper closed convex and lower bounded, del f(i) is Lipschitz continuous with constant L-fi > 0, and the stepsize a is fixed. Provided that alpha <= min{(1 + lambda(n)) (W))/L-h, 1/L-(f) over bar}, where L-h = max(i){L-fi} and L-(f) over bar = (1)(n) Sigma(n)(i=1) L-fi, the objective errors of all the local solutions and the networkwide mean solution reduce at rates of O(1/k) until they reach a level of O(alpha). If fi are strongly convex with modulus mu(fi), and alpha <= min{(1+lambda(n)(W))/L-h, 1/(L-f+mu(f))}, where mu(f) - 1/n Sigma(n)(i=1) mu(fi), then all the local solutions and the mean solution converge to the global minimizer x* at a linear rate until reaching an O(alpha)-neighborhood of x*. We also develop an iteration for decentralized basis pursuit and establish its linear convergence to an O(alpha)-neighborhood of the true sparse signal. This analysis reveals how the convergence of x((i))(k + 1) <- Sigma(n)(j=1) wijx(j)(k) - alpha del f(i)(x((i))(k)), for each agent i, depends on the stepsize, function convexity, and network spectrum.
Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

Zhengxue Li,Wei Wu,Yulong Tian

DOI: https://doi.org/10.1016/j.cam.2003.08.062

IF: 2.872

2004-01-01

Journal of Computational and Applied Mathematics

Abstract:In this paper, we study the convergence of an online gradient method for feed-forward neural networks. The input training examples are permuted stochastically in each cycle of iteration. A monotonicity and a weak convergence of deterministic nature are proved.

Convergence of gradient descent algorithm for Pi-sigma neural networks

Convergence of online gradient method for pi-sigma neural networks

Convergence of online gradient methods for Pi-Sigma neural network with a penalty term

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Convergence Analysis of Batch Gradient Algorithm for Three Classes of Sigma-Pi Neural Networks

Training Pi-Sigma Network by Online Gradient Algorithm with Penalty for Small Weight Update.

Convergence of Batch Gradient Learning Algorithm with Smoothing L1/2 Regularization for Sigma–Pi–Sigma Neural Networks

A Modified Higher-Order Feed Forward Neural Network with Smoothing Regularization

A Modifled Gradient-Based Neuro-Fuzzy Learning Algorithm for Pi-Sigma Network Based on First-Order Takagi-Sugeno System

Convergence of gradient method for double parallel feedforward neural network

A new Sigma-Pi-Sigma neural network based on $ L_1 $ and $ L_2 $ regularization and applications

Convergence of Stochastic Gradient Descent in Deep Neural Network

An improved asynchronous batch gradient method for ridge polynomial neural network

Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks

Convergence and monotonicity of an online gradient method with penalty for neural networks

On the Convergence and Improvement of Stochastic Normalized Gradient Descent

On Faster Convergence of Scaled Sign Gradient Descent

A Polak-Ribière-Polyak Conjugate Gradient-Based Neuro-Fuzzy Network and Its Convergence

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

On the Convergence of Decentralized Gradient Descent.

Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs