Abstract:Stability arguments are often used to prevent learning algorithms from having ever increasing activity and weights that hinder generalization. However, stability conditions can clash with the sparsity required to augment the energy efficiency of spiking neurons. Nonetheless it can also provide solutions. In fact, spiking Neuromorphic Computing uses binary activity to improve Artificial Intelligence energy efficiency. However, its non-smoothness requires approximate gradients, known as Surrogate Gradients (SG), to close the performance gap with Deep Learning. Several SG have been proposed in the literature, but it remains unclear how to determine the best SG for a given task and network. Thus, we aim at theoretically define the best SG, through stability arguments, to reduce the need for grid search. In fact, we show that more complex tasks and networks need more careful choice of SG, even if overall the derivative of the fast sigmoid tends to outperform the other, for a wide range of learning rates. We therefore design a stability based theoretical method to choose initialization and SG shape before training on the most common spiking neuron, the Leaky Integrate and Fire (LIF). Since our stability method suggests the use of high firing rates at initialization, which is non-standard in the neuromorphic literature, we show that high initial firing rates, combined with a sparsity encouraging loss term introduced gradually, can lead to better generalization, depending on the SG shape. Our stability based theoretical solution, finds a SG and initialization that experimentally result in improved accuracy. We show how it can be used to reduce the need of extensive grid-search of dampening, sharpness and tail-fatness of the SG. We also show that our stability concepts can be extended to be applicable on different LIF variants, such as DECOLLE and fluctuations-driven initializations.

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Surrogate Gradient Scaling for Directly Training Spiking Neural Networks.

IM-LIF: Improved Neuronal Dynamics with Attention Mechanism for Direct Training Deep Spiking Neural Network

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks

Directly Training Temporal Spiking Neural Network with Sparse Surrogate Gradient

Efficient Surrogate Gradients for Training Spiking Neural Networks

Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Gradual Surrogate Gradient Learning in Deep Spiking Neural Networks

Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks.

Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks

Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks

Membrane Potential Distribution Adjustment and Parametric Surrogate Gradient in Spiking Neural Networks

A progressive surrogate gradient learning for memristive spiking neural network

Reasonable Gradients for Online Training Algorithms in Spiking Neural Networks

KLIF: An Optimized Spiking Neuron Unit for Tuning Surrogate Gradient Function

Stabilizing Spiking Neuron Training

Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning