Abstract:We investigate a new approach to compute the gradients of artificial neural networks (ANNs), based on the so-called push-out likelihood ratio method. Unlike the widely used backpropagation (BP) method that requires continuity of the loss function and the activation function, our approach bypasses this requirement by injecting artificial noises into the signals passed along the neurons. We show how this approach has a similar computational complexity as BP, and moreover is more advantageous in terms of removing the backward recursion and eliciting transparent formulas. We also formalize the connection between BP, a pivotal technique for training ANNs, and infinitesimal perturbation analysis, a classic path-wise derivative estimation approach, so that both our new proposed methods and BP can be better understood in the context of stochastic gradient estimation. Our approach allows efficient training for ANNs with more flexibility on the loss and activation functions, and shows empirical improvements on the robustness of ANNs under adversarial attacks and corruptions of natural noises. Summary of Contribution: Stochastic gradient estimation has been studied actively in simulation for decades and becomes more important in the era of machine learning and artificial intelligence. The stochastic gradient descent is a standard technique for training the artificial neural networks (ANNs), a pivotal problem in deep learning. The most popular stochastic gradient estimation technique is the backpropagation method. We find that the backpropagation method lies in the family of infinitesimal perturbation analysis, a path-wise gradient estimation technique in simulation. Moreover, we develop a new likelihood ratio-based method, another popular family of gradient estimation technique in simulation, for training more general ANNs, and demonstrate that the new training method can improve the robustness of the ANN.

Training Artificial Neural Networks by Generalized Likelihood Ratio Method: an Effective Way to Improve Robustness.

A New Likelihood Ratio Method for Training Artificial Neural Networks

A Novel Noise Injection-based Training Scheme for Better Model Robustness

Adaptive Retraining for Neural Network Robustness in Classification

Towards Robust Training of Neural Networks by Regularizing Adversarial Gradients

One Forward is Enough for Neural Network Training via Likelihood Ratio Method

Robustness Enhancement in Neural Networks with Alpha-Stable Training Noise

Improving Model Robustness Against Adversarial Examples with Redundant Fully Connected Layer.

Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness

Training Robust Deep Neural Networks via Adversarial Noise Propagation

Robust Stability and Robust Periodicity of Delayed Recurrent Neural Networks with Noise Disturbance

Activity-dependent Neuron Model for Noise Resistance

An Empirical Study on the Effect of Training Data Perturbations on Neural Network Robustness

Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Training

Robust Training of Neural Networks at Arbitrary Precision and Sparsity

Towards Certifying L Robustness Using Neural Networks with L-Dist Neurons

Noise Optimization in Artificial Neural Networks

Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction

Network Robustness Prediction: Influence of Training Data Distributions

It begins with a boundary: A geometric view on probabilistically robust learning

Robust noise-aware algorithm for randomized neural network and its convergence properties