Abstract:Spiking neural networks (SNNs) are increasingly applied to deep architectures. Recent works are developed to apply spatio-temporal backpropagation to directly train deep SNNs. But the binary and non-differentiable properties of spike activities force directly trained SNNs to suffer from serious gradient vanishing. In this paper, we first analyze the cause of the gradient vanishing problem and identify that the gradients mostly backpropagate along the synaptic currents. Based on that, we modify the synaptic current equation of leaky-integrate-fire neuron model and propose the improved LIF (IM-LIF) neuron model on the basis of the temporal-wise attention mechanism. We utilize the temporal-wise attention mechanism to selectively establish the connection between the current and historical response values, which can empirically enable the neuronal states to update resilient to the gradient vanishing problem. Furthermore, to capture the neuronal dynamics embedded in the output incorporating the IM-LIF model, we present a new temporal loss function to constrain the output of the network close to the target distribution. The proposed new temporal loss function could not only act as a regularizer to eliminate output outliers, but also assign the network loss credit to the voltage at a specific time point. Then we modify the ResNet and VGG architecture based on the IM-LIF model to build deep SNNs. We evaluate our work on image datasets and neuromorphic datasets. Experimental results and analysis show that our method can help build deep SNNs with competitive performance in both accuracy and latency, including 95.66% on CIFAR-10, 77.42% on CIFAR-100, 55.37% on Tiny-ImageNet, 97.33% on DVS-Gesture, and 80.50% on CIFAR-DVS with very few timesteps.

Membrane Potential Distribution Adjustment and Parametric Surrogate Gradient in Spiking Neural Networks

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Efficient Surrogate Gradients for Training Spiking Neural Networks

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

Surrogate Gradient Scaling for Directly Training Spiking Neural Networks.

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Directly Training Temporal Spiking Neural Network with Sparse Surrogate Gradient

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks

Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks.

Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks

A progressive surrogate gradient learning for memristive spiking neural network

Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks

Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks

Gradual Surrogate Gradient Learning in Deep Spiking Neural Networks

A Progressive Surrogate Gradient Learning for Memristive SNN

A Dynamic Surrogate Gradient Function for Memristive Spiking Neural Networks

IM-LIF: Improved Neuronal Dynamics with Attention Mechanism for Direct Training Deep Spiking Neural Network

Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning

Comprehensive Online Training and Deployment for Spiking Neural Networks

An Improved Probabilistic Spiking Neural Network with Enhanced Discriminative Ability