Abstract:AbstractMinimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L"2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing L"1"/"2 regularization term. For normal L"1"/"2 regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal L"1"/"2 regularization term can be addressed. This paper shows the strong convergence results for the smoothing L"1"/"2 regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L"2 and normal L"1"/"2 regularizations respectively.

Convergence and monotonicity of an online gradient method with penalty for neural networks

Boundedness and Convergence of Online Gradient Method with Penalty and Momentum

Boundedness and Convergence of Online Gradient Method with Penalty for Linear Output Feedforward Neural Networks

Convergence of A Gradient Algorithm with Penalty for Training Two-Layer Neural Networks

Convergence of online gradient methods for Pi-Sigma neural network with a penalty term

Convergence of Online Gradient Method with a Penalty Term for BP Neural Network with Stochastic Inputs

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Convergence of an online gradient method with inner-product penalty and adaptive momentum

Online Gradient Methods with a Punishing Term for Neural Networks

Convergence of batch BP algorithm with penalty for FNN training

Convergence Analysis of Multilayer Feedforward Networks Trained with Penalty Terms: A review

Strong Convergence Of Gradient Methods For Bp Networks Training

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Convergence Analysis of Inverse Iterative Neural Networks with L2 Penalty

Boundedness and Convergence of Batch Back-Propagation Algorithm with Penalty for Feedforward Neural Networks

Convergence Analysis of Online Gradient Method for BP Neural Networks

Boundedness and Convergence of Split-Complex Back-Propagation Algorithm with Momentum and Penalty

Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Convergence of On-Line Gradient Methods for Two-Layer Feedforward Neural Networks