Abstract:Numerous experimental studies suggest that noise is inherent in the human brain. However, the functional importance of noise remains unknown. n particular, from a computational perspective, such stochasticity is potentially harmful to brain function. In machine learning, a large number of saddle points are surrounded by high error plateaus and give the illusion of the existence of local minimum. As a result, being trapped in the saddle points can dramatically impair learning and adding noise will attack such saddle point problems in high-dimensional optimization, especially under the strict saddle condition. Motivated by these arguments, we propose one biologically plausible noise structure and demonstrate that noise can efficiently improve the optimization performance of spiking neural networks based on stochastic gradient descent. The strict saddle condition for synaptic plasticity is deduced, and under such conditions, noise can help optimization escape from saddle points on high dimensional domains. The theoretical results explain the stochasticity of synapses and guide us on how to make use of noise. In addition, we provide biological interpretations of proposed noise structures from two points: one based on the free energy principle in neuroscience and another based on observations of in vivo experiments. Our simulation results manifest that in the learning and test phase, the accuracy of synaptic sampling with noise is almost 20% higher than that without noise for synthesis dataset, and the gain in accuracy with/without noise is at least 10% for the MNIST and CIFAR-10 dataset. Our study provides a new learning framework for the brain and sheds new light on deep noisy spiking neural networks.

Role of Noises in Neural Networks

Effect of noises on two-layer hodgkin-huxley neuronal network

Coherent noise enables probabilistic sequence replay in spiking neuronal networks

Exploiting Noise As a Resource for Computation and Learning in Spiking Neural Networks

Noise Facilitation in Associative Memories of Exponential Capacity

How noise affects memory in linear recurrent networks

Activity-dependent Neuron Model for Noise Resistance

Long-term memory stabilized by noise-induced rehearsal

How Neuronal Noises Influence the Spiking Neural Networks’s Cognitive Learning Process: A Preliminary Study

Noise Helps Optimization Escape from Saddle Points in the Neural Dynamics.

Training neural networks with structured noise improves classification and generalization

Functional importance of noise in neuronal information processing

Implications of Noise in Resistive Memory on Deep Neural Networks for Image Classification

Noise Helps Optimization Escape from Saddle Points in the Synaptic Plasticity

Using noise to probe recurrent neural network structure and prune synapses

Noise-induced memory in extended excitable systems

Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones

Noise Optimization in Artificial Neural Networks

Supervised Learning in Spiking Neural Networks with Noise-Threshold

Towards Understanding the Importance of Noise in Training Neural Networks