Abstract:In this paper, a new variant of feedforward neural networks has been proposed for a class of nonsmooth optimization problems. The penalty term of the presented neural networks stems from the Group Lasso method which selects hidden variables in a grouped manner. To deal with the non-differentiability of the original penalty term (l1−l2 norm) and avoid oscillations, smoothing techniques have been used to approximate the objective function. It is assumed that the training samples are supplied to the networks in a specific incremental way during training, that is, in each cycle samples are supplied in a fixed order. Then, under suitable assumptions on learning rate, penalization coefficients and smoothing parameters, the weak and strong convergence of the training process for the smoothing neural networks have been proved. The convergence analysis shows that the gradient of the smoothing error function approaches zero and the weight sequence converges to a fixed point, respectively. We demonstrate how the smoothing approximation parameter can be updated in the training procedure so as to guarantee the convergence of the procedure to a Clarke stationary point of the original optimization problem. In addition, we have proved that the original nonsmoothing algorithm with l1−l2 norm penalty converges consistently to the same optimum solution with the corresponding smoothed algorithm. Numerical simulations demonstrate the convergence and effectiveness of the proposed training algorithm.

Batch Gradient Training Method with Smoothing Group L_0 Regularization for Feedfoward Neural Networks

Batch Gradient Training Method with Smoothing Group <i>L</i><sub>0</sub> Regularization for Feedfoward Neural Networks

Batch gradient training method with smoothing ℓ_ 0 regularization for feedforward neural networks

Batch gradient training method with smoothing $$\\boldsymbol{\\ell}_{\\bf 0}$$ℓ0 regularization for feedforward neural networks

Batch gradient method with smoothing L1/2 regularization for training of feedforward neural networks.

Online Gradient Method with Smoothing ℓ 0 Regularization for Feedforward Neural Networks

Batch Gradient Neuro-Fuzzy Learning Method with Smoothing L0 Regularization for the First-Order Takagi-Sugeno System

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

L0 image smoothing via iterating truncated L1 gradient regularization

A Novel Pruning Algorithm for Smoothing Feedforward Neural Networks Based on Group Lasso Method

Convergence of Batch Gradient Learning with Smoothing Regularization and Adaptive Momentum for Neural Networks

Convergence Analyses on Sparse Feedforward Neural Networks Via Group Lasso Regularization.

Convergence of Batch Gradient Learning Algorithm with Smoothing L1/2 Regularization for Sigma–Pi–Sigma Neural Networks

A Modified Higher-Order Feed Forward Neural Network with Smoothing Regularization

A New Conjugate Gradient Method with Smoothing L_1/2 Regularization Based on a Modified Secant Equation for Training Neural Networks

Learning Broad Learning System with Controllable Sparsity Through L0 Regularization

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Learning Sparse Neural Networks through L0 Regularization

A New Conjugate Gradient Method with Smoothing $$L_{1/2} $$L1/2 Regularization Based on a Modified Secant Equation for Training Neural Networks

Smooth Group L1/2 Regularization for Input Layer of Feedforward Neural Networks

Group $L_{1/2}$ Regularization for Pruning Hidden Layer Nodes of Feedforward Neural Networks