Abstract:Physical neural networks are promising candidates for next generation artificial intelligence hardware. In such architectures, neurons and connections are physically realized and do not leverage digital concepts with their practically infinite signal-to-noise ratio to encode, transduce and transform information. They therefore are prone to noise with a variety of statistical and architectural properties, and effective strategies leveraging network-inherent assets to mitigate noise in an hardware-efficient manner are important in the pursuit of next generation neural network hardware. Based on analytical derivations, we here introduce and analyse a variety of different noise-mitigation approaches. We analytically show that intra-layer connections in which the connection matrix's squared mean exceeds the mean of its square fully suppresses uncorrelated noise. We go beyond and develop two synergistic strategies for noise that is uncorrelated and correlated across populations of neurons. First, we introduce the concept of ghost neurons, where each group of neurons perturbed by correlated noise has a negative connection to a single neuron, yet without receiving any input information. Secondly, we show that pooling of neuron populations is an efficient approach to suppress uncorrelated noise. As such, we developed a general noise mitigation strategy leveraging the statistical properties of the different noise terms most relevant in analogue hardware. Finally, we demonstrate the effectiveness of this combined approach for trained neural network classifying the MNIST handwritten digits, for which we achieve a 4-fold improvement of the output signal-to-noise ratio and increase the classification accuracy almost to the level of the noise-free network.

Understanding and mitigating noise in trained deep neural networks

Fundamental aspects of noise in analog-hardware neural networks

Noise mitigation strategies in physical feedforward neural networks

Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation

Exploiting Noise As a Resource for Computation and Learning in Spiking Neural Networks

Noisy Computations during Inference: Harmful or Helpful?

Activity-dependent Neuron Model for Noise Resistance

Impact of white noise in artificial neural networks trained for classification: performance and noise mitigation strategies

Improving Analog Neural Network Robustness: A Noise-Agnostic Approach with Explainable Regularizations

Noise impact on recurrent neural network with linear activation function

Training Recurrent Neural Networks against Noisy Computations during Inference

Walking Noise: On Layer-Specific Robustness of Neural Architectures against Noisy Computations and Associated Characteristic Learning Dynamics

Improving the Robustness of Analog Deep Neural Networks Through a Bayes-optimized Noise Injection Approach

Efficient Noise Mitigation for Enhancing Inference Accuracy in DNNs on Mixed-Signal Accelerators

Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models

Critical initialisation for deep signal propagation in noisy rectifier neural networks

Noise-Resilient Designs for Optical Neural Networks

Neural Dynamics for Improving Optimiser in Deep Learning with Noise Considered

Noise Optimization in Artificial Neural Networks

Enhance DNN Adversarial Robustness and Efficiency via Injecting Noise to Non-Essential Neurons

Noise Helps Optimization Escape from Saddle Points in the Synaptic Plasticity