Abstract:AbstractMinimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L"2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing L"1"/"2 regularization term. For normal L"1"/"2 regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal L"1"/"2 regularization term can be addressed. This paper shows the strong convergence results for the smoothing L"1"/"2 regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L"2 and normal L"1"/"2 regularizations respectively.

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs

Convergence and monotonicity of an online gradient method with penalty for neural networks

Convergence of Online Gradient Method with a Penalty Term for BP Neural Network with Stochastic Inputs

Boundedness and Convergence of Online Gradient Method with Penalty for Linear Output Feedforward Neural Networks

Convergence of an online gradient method with inner-product penalty and adaptive momentum

Boundedness and Convergence of Online Gradient Method with Penalty and Momentum

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Online Gradient Methods with a Punishing Term for Neural Networks

Convergence of batch gradient algorithm for feedforward neural network training

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Convergence of online gradient methods for Pi-Sigma neural network with a penalty term

Convergence of Online Gradient Method for Double Parallel Feedforward Neural Network

Convergence Analysis of Online Gradient Method for BP Neural Networks

Strong Convergence Of Gradient Methods For Bp Networks Training

A Class of Gradient Algorithms with Variable Learning Rates and Convergence Analysis for Feedforward Neural Networks Training

Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks

Convergence of A Gradient Algorithm with Penalty for Training Two-Layer Neural Networks

Convergence of Gradient Method for A Fully Recurrent Neural Network

Boundedness and Convergence of Batch Back-Propagation Algorithm with Penalty for Feedforward Neural Networks