Abstract:Stability arguments are often used to prevent learning algorithms from having ever increasing activity and weights that hinder generalization. However, stability conditions can clash with the sparsity required to augment the energy efficiency of spiking neurons. Nonetheless it can also provide solutions. In fact, spiking Neuromorphic Computing uses binary activity to improve Artificial Intelligence energy efficiency. However, its non-smoothness requires approximate gradients, known as Surrogate Gradients (SG), to close the performance gap with Deep Learning. Several SG have been proposed in the literature, but it remains unclear how to determine the best SG for a given task and network. Thus, we aim at theoretically define the best SG, through stability arguments, to reduce the need for grid search. In fact, we show that more complex tasks and networks need more careful choice of SG, even if overall the derivative of the fast sigmoid tends to outperform the other, for a wide range of learning rates. We therefore design a stability based theoretical method to choose initialization and SG shape before training on the most common spiking neuron, the Leaky Integrate and Fire (LIF). Since our stability method suggests the use of high firing rates at initialization, which is non-standard in the neuromorphic literature, we show that high initial firing rates, combined with a sparsity encouraging loss term introduced gradually, can lead to better generalization, depending on the SG shape. Our stability based theoretical solution, finds a SG and initialization that experimentally result in improved accuracy. We show how it can be used to reduce the need of extensive grid-search of dampening, sharpness and tail-fatness of the SG. We also show that our stability concepts can be extended to be applicable on different LIF variants, such as DECOLLE and fluctuations-driven initializations.

Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoders

Toward Efficient Processing and Learning with Spikes: New Approaches for Multispike Learning

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Stabilizing Spiking Neuron Training

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

Supervised Learning with First-to-Spike Decoding in Multilayer Spiking Neural Networks

High-performance deep spiking neural networks with 0.3 spikes per neuron

Smooth Exact Gradient Descent Learning in Spiking Neural Networks

Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks

SuperSpike: Supervised learning in multi-layer spiking neural networks

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks

Learning Precise Spike Train to Spike Train Transformations in Multilayer Feedforward Neuronal Networks

Spike-based computation using classical recurrent neural networks

The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks

A Synapse-Threshold Synergistic Learning Approach for Spiking Neural Networks

Spiking representation learning for associative memories

SpikePropamine: Differentiable Plasticity in Spiking Neural Networks