Abstract:We investigate a new approach to compute the gradients of artificial neural networks (ANNs), based on the so-called push-out likelihood ratio method. Unlike the widely used backpropagation (BP) method that requires continuity of the loss function and the activation function, our approach bypasses this requirement by injecting artificial noises into the signals passed along the neurons. We show how this approach has a similar computational complexity as BP, and moreover is more advantageous in terms of removing the backward recursion and eliciting transparent formulas. We also formalize the connection between BP, a pivotal technique for training ANNs, and infinitesimal perturbation analysis, a classic path-wise derivative estimation approach, so that both our new proposed methods and BP can be better understood in the context of stochastic gradient estimation. Our approach allows efficient training for ANNs with more flexibility on the loss and activation functions, and shows empirical improvements on the robustness of ANNs under adversarial attacks and corruptions of natural noises. Summary of Contribution: Stochastic gradient estimation has been studied actively in simulation for decades and becomes more important in the era of machine learning and artificial intelligence. The stochastic gradient descent is a standard technique for training the artificial neural networks (ANNs), a pivotal problem in deep learning. The most popular stochastic gradient estimation technique is the backpropagation method. We find that the backpropagation method lies in the family of infinitesimal perturbation analysis, a path-wise gradient estimation technique in simulation. Moreover, we develop a new likelihood ratio-based method, another popular family of gradient estimation technique in simulation, for training more general ANNs, and demonstrate that the new training method can improve the robustness of the ANN.

Padé Rational Neural Network and Its Weights-Direct-Determination Method

Fourier Neural Network Circuit Implementation Based on Direct Weight Determination

A Method for Computing Inverse Parametric PDE Problems with Random-Weight Neural Networks

A Priori Estimation of the Approximation, Optimization and Generalization Errors of Random Neural Networks for Solving Partial Differential Equations

Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

Power-Enhanced Residual Network for Function Approximation and Physics-Informed Inverse Problems

Efficient Design of Neural Networks with Random Weights

A Pseudoinversion-Free Method for Weight Updating in Broad Learning System

A Gegenbauer Neural Network with Regularized Weights Direct Determination for Classification

Self-adaptive deep neural network: Numerical approximation to functions and PDEs

Predict globally, correct locally: Parallel-in-time optimization of neural networks

An approximation by neural networkswith a fixed weight

A Priori Error Estimation of Physics-Informed Neural Networks Solving Allen--Cahn and Cahn--Hilliard Equations

A New Likelihood Ratio Method for Training Artificial Neural Networks

Nonlinear Weighted Directed Acyclic Graph and A Priori Estimates for Neural Networks

Determining the Weights of A Fourier Series Neural Network on the Basis of the Multidimensional Discrete Fourier Transform

A New Method for Decision on the Structure of RBF Neural Network

A Rough Set Approach to BP Neural Network Designing

Physical Information Neural Networks for Solving High-index Differential-algebraic Equation Systems Based on Radau Methods

Back Propagation With Randomized Cost Function For Training Neural Networks

Principal Component Analysis Based Probability Neural Network Optimization