Abstract:Stability arguments are often used to prevent learning algorithms from having ever increasing activity and weights that hinder generalization. However, stability conditions can clash with the sparsity required to augment the energy efficiency of spiking neurons. Nonetheless it can also provide solutions. In fact, spiking Neuromorphic Computing uses binary activity to improve Artificial Intelligence energy efficiency. However, its non-smoothness requires approximate gradients, known as Surrogate Gradients (SG), to close the performance gap with Deep Learning. Several SG have been proposed in the literature, but it remains unclear how to determine the best SG for a given task and network. Thus, we aim at theoretically define the best SG, through stability arguments, to reduce the need for grid search. In fact, we show that more complex tasks and networks need more careful choice of SG, even if overall the derivative of the fast sigmoid tends to outperform the other, for a wide range of learning rates. We therefore design a stability based theoretical method to choose initialization and SG shape before training on the most common spiking neuron, the Leaky Integrate and Fire (LIF). Since our stability method suggests the use of high firing rates at initialization, which is non-standard in the neuromorphic literature, we show that high initial firing rates, combined with a sparsity encouraging loss term introduced gradually, can lead to better generalization, depending on the SG shape. Our stability based theoretical solution, finds a SG and initialization that experimentally result in improved accuracy. We show how it can be used to reduce the need of extensive grid-search of dampening, sharpness and tail-fatness of the SG. We also show that our stability concepts can be extended to be applicable on different LIF variants, such as DECOLLE and fluctuations-driven initializations.

Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks

Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

Surrogate Gradient Scaling for Directly Training Spiking Neural Networks.

Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks

Efficient Surrogate Gradients for Training Spiking Neural Networks

Smooth Exact Gradient Descent Learning in Spiking Neural Networks

Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks.

Directly Training Temporal Spiking Neural Network with Sparse Surrogate Gradient

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Membrane Potential Distribution Adjustment and Parametric Surrogate Gradient in Spiking Neural Networks

A generalized neural tangent kernel for surrogate gradient learning

Stabilizing Spiking Neuron Training

Gradual Surrogate Gradient Learning in Deep Spiking Neural Networks

Gradient Descent for Spiking Neural Networks

Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Toward spike-based stochastic neural computing

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks