Abstract:Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes) which can potentially lead to higher energy-efficiency in neuromorphic hardware implementations. Many works have shown that an SNN for inference can be formed by copying the weights from a trained Artificial Neural Network (ANN) and setting the firing threshold for each layer as the maximum input received in that layer. These type of converted SNNs require a large number of time steps to achieve competitive accuracy which diminishes the energy savings. The number of time steps can be reduced by training SNNs with spike-based backpropagation from scratch, but that is computationally expensive and slow. To address these challenges, we present a computationally-efficient training technique for deep SNNs. We propose a hybrid training methodology: 1) take a converted SNN and use its weights and thresholds as an initialization step for spike-based backpropagation, and 2) perform incremental spike-timing dependent backpropagation (STDB) on this carefully initialized network to obtain an SNN that converges within few epochs and requires fewer time steps for input processing. STDB is performed with a novel surrogate gradient function defined using neuron's spike time. The proposed training methodology converges in less than 20 epochs of spike-based backpropagation for most standard image classification datasets, thereby greatly reducing the training complexity compared to training SNNs from scratch. We perform experiments on CIFAR-10, CIFAR-100, and ImageNet datasets for both VGG and ResNet architectures. We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time steps, which is 10X faster compared to converted SNNs with similar accuracy.

Spiking Deep Residual Network

Spiking Deep Residual Networks.

Advancing Spiking Neural Networks Toward Deep Residual Learning

BitSNNs: Revisiting Energy-efficient Spiking Neural Networks

Deep Residual Learning in Spiking Neural Networks

Spike Trains Encoding and Threshold Rescaling Method for Deep Spiking Neural Networks

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Multi-Level Firing with Spiking DS-ResNet: Enabling Better and Deeper Directly-Trained Spiking Neural Networks

Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Advancing Deep Residual Learning by Solving the Crux of Degradation in Spiking Neural Networks

Deep Spiking Neural Network Using Spatio-temporal Backpropagation with Variable Resistance.

Effective Active Learning Method for Spiking Neural Networks.

Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization

Training much deeper spiking neural networks with a small number of time-steps

Bio-Inspired Deep Spiking Neural Network For Image Classification

Going Deeper With Directly-Trained Larger Spiking Neural Networks