Abstract:Numerous experimental studies suggest that noise is inherent in the human brain. However, the functional importance of noise remains unknown. n particular, from a computational perspective, such stochasticity is potentially harmful to brain function. In machine learning, a large number of saddle points are surrounded by high error plateaus and give the illusion of the existence of local minimum. As a result, being trapped in the saddle points can dramatically impair learning and adding noise will attack such saddle point problems in high-dimensional optimization, especially under the strict saddle condition. Motivated by these arguments, we propose one biologically plausible noise structure and demonstrate that noise can efficiently improve the optimization performance of spiking neural networks based on stochastic gradient descent. The strict saddle condition for synaptic plasticity is deduced, and under such conditions, noise can help optimization escape from saddle points on high dimensional domains. The theoretical results explain the stochasticity of synapses and guide us on how to make use of noise. In addition, we provide biological interpretations of proposed noise structures from two points: one based on the free energy principle in neuroscience and another based on observations of in vivo experiments. Our simulation results manifest that in the learning and test phase, the accuracy of synaptic sampling with noise is almost 20% higher than that without noise for synthesis dataset, and the gain in accuracy with/without noise is at least 10% for the MNIST and CIFAR-10 dataset. Our study provides a new learning framework for the brain and sheds new light on deep noisy spiking neural networks.

Noise Optimization in Artificial Neural Networks

Fooling Neural Network Interpretations - Adversarial Noise to Attack Images.

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

Activity-dependent Neuron Model for Noise Resistance

Exploiting Noise As a Resource for Computation and Learning in Spiking Neural Networks

Noise Helps Optimization Escape from Saddle Points in the Synaptic Plasticity

A Novel Noise Injection-based Training Scheme for Better Model Robustness

Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack

Enhance DNN Adversarial Robustness and Efficiency via Injecting Noise to Non-Essential Neurons

Noise Helps Optimization Escape from Saddle Points in the Neural Dynamics.

A Noise-Sensitivity-Analysis-Based Test Prioritization Technique for Deep Neural Networks

Neural-network-based Optimization and Analysis for Nonlinear Stochastic Systems.

Neural Dynamics for Improving Optimiser in Deep Learning with Noise Considered

Robust Stability and Robust Periodicity of Delayed Recurrent Neural Networks with Noise Disturbance

Towards Understanding the Importance of Noise in Training Neural Networks

Improving the Robustness of Analog Deep Neural Networks Through a Bayes-optimized Noise Injection Approach

Robustness Enhancement in Neural Networks with Alpha-Stable Training Noise

Improving Analog Neural Network Robustness: A Noise-Agnostic Approach with Explainable Regularizations

Non-informative Noise-Enhanced Stochastic Neural Networks for Improving Adversarial Robustness

Algorithm and Application of a Noise-Tolerant Neural Network

Understanding and mitigating noise in trained deep neural networks