Abstract:Spiking neural networks (SNNs) have increasingly drawn massive research attention due to biological interpretability and efficient computation. Recent achievements are devoted to utilizing the surrogate gradient (SG) method to avoid the dilemma of non-differentiability of spiking activity to directly train SNNs by backpropagation. However, the fixed width of the SG leads to gradient vanishing and mismatch problems, thus limiting the performance of directly trained SNNs. In this work, we propose a novel perspective to unlock the width limitation of SG, called the learnable surrogate gradient (LSG) method. The LSG method modulates the width of SG according to the change of the distribution of the membrane potentials, which is identified to be related to the decay factors based on our theoretical analysis. Then we introduce the trainable decay factors to implement the LSG method, which can optimize the width of SG automatically during training to avoid the gradient vanishing and mismatch problems caused by the limited width of SG. We evaluate the proposed LSG method on both image and neuromorphic datasets. Experimental results show that the LSG method can effectively alleviate the blocking of gradient propagation caused by the limited width of SG when training deep SNNs directly. Meanwhile, the LSG method can help SNNs achieve competitive performance on both latency and accuracy.

Towards Efficient On-Chip Learning for Spiking Neural Networks Accelerator with Surrogate Gradient

A Reconfigurable FPGA-based Spiking Neural Network Accelerator

Spike Trains Encoding Optimization for Spiking Neural Networks Implementation in FPGA

A Low Power and Low Latency FPGA-Based Spiking Neural Network Accelerator

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

A low cost neuromorphic learning engine based on a high performance supervised SNN learning algorithm

An Event-driven Spiking Neural Network Accelerator with On-chip Sparse Weight

A Fast Spiking Neural Network Accelerator based on BP-STDP Algorithm and Weighted Neuron Model

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

An Efficient Spiking Neural Network Accelerator with Sparse Weight.

SPAT: FPGA-based Sparsity-Optimized Spiking Neural Network Training Accelerator with Temporal Parallel Dataflow

High-speed, low-power, and configurable on-chip training acceleration platform for spiking neural networks

Enabling Efficient On-Edge Spiking Neural Network Acceleration with Highly Flexible FPGA Architectures

A 28nm Configurable Asynchronous SNN Accelerator with Energy-Efficient Learning

Fine-Tuning Surrogate Gradient Learning for Optimal Hardware Performance in Spiking Neural Networks

A 0.67-to-5.4 TSOPs/W Spiking Neural Network Accelerator with 128/256 Reconfigurable Neurons and Asynchronous Fully Connected Synapses

A Convolutional Spiking Neural Network Accelerator with the Sparsity-Aware Memory and Compressed Weights

A 67.5μJ/Prediction Accelerator for Spiking Neural Networks in Image Segmentation

Highly efficient neuromorphic learning system of spiking neural network with multi-compartment leaky integrate-and-fire neurons

Hardware-Software Co-optimised Fast and Accurate Deep Reconfigurable Spiking Inference Accelerator Architecture Design Methodology