Abstract:Spiking Neural Networks (SNNs) have emerged as a hardware efficient architecture for classification tasks. The challenge of spike-based encoding has been the lack of a universal training mechanism performed entirely using spikes. There have been several attempts to adopt the powerful backpropagation (BP) technique used in non-spiking artificial neural networks (ANN): (1) SNNs can be trained by externally computed numerical gradients. (2) A major advancement towards native spike-based learning has been the use of approximate Backpropagation using spike-time dependent plasticity (STDP) with phased forward/backward passes. However, the transfer of information between such phases for gradient and weight update calculation necessitates external memory and computational access. This is a challenge for standard neuromorphic hardware implementations. In this paper, we propose a stochastic SNN based Back-Prop (SSNN-BP) algorithm that utilizes a composite neuron to simultaneously compute the forward pass activations and backward pass gradients explicitly with spikes. Although signed gradient values are a challenge for spike-based representation, we tackle this by splitting the gradient signal into positive and negative streams. The composite neuron encodes information in the form of stochastic spike-trains and converts Backpropagation weight updates into temporally and spatially local spike coincidence updates compatible with hardware-friendly Resistive Processing Units (RPUs). Furthermore, we characterize the quantization effect of discrete spike-based weight update to show that our method approaches BP ANN baseline with sufficiently long spike-trains. Finally, we show that the well-performing softmax cross-entropy loss function can be implemented through inhibitory lateral connections enforcing a Winner Take All (WTA) rule. Our SNN with a 2-layer network shows excellent generalization through comparable performance to ANNs with equivalent architecture and regularization parameters on static image datasets like MNIST, Fashion-MNIST, Extended MNIST, and temporally encoded image datasets like Neuromorphic MNIST datasets. Thus, SSNN-BP enables BP compatible with purely spike-based neuromorphic hardware.

Direct Training of Hardware-Friendly Weight Binarized Spiking Neural Network with Surrogate Gradient Learning Towards Spatio-Temporal Event-Based Dynamic Data Recognition

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

A Sparsity-Adapted Hardware Implementation of SNN for Cortical Spike Trains Decoding

A temporally and spatially local spike-based backpropagation algorithm to enable training in hardware

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Deep Spiking Neural Networks with Binary Weights for Object Recognition

Batch Normalization-Free Weight-Binarized SNN Based on Hardware-Saving IF Neuron.

NBSSN: A Neuromorphic Binary Single-Spike Neural Network for Efficient Edge Intelligence.

A Scatter-and-Gather Spiking Convolutional Neural Network on a Reconfigurable Neuromorphic Hardware

A Neuromorphic-Hardware Oriented Bio-Plausible Online-Learning Spiking Neural Network Model.

Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

Digital Implementation of the Spiking Neural Network and Its Digit Recognition

Scalable NoC-based Neuromorphic Hardware Learning and Inference

Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performance.

Directly Training Temporal Spiking Neural Network with Sparse Surrogate Gradient

ALBSNN: ultra-low latency adaptive local binary spiking neural network with accuracy loss estimator

ACE-SNN: Algorithm-Hardware Co-design of Energy-Efficient & Low-Latency Deep Spiking Neural Networks for 3D Image Recognition

Ultra-low Latency Spiking Neural Networks with Spatio-Temporal Compression and Synaptic Convolutional Block

VTSNN: a virtual temporal spiking neural network

Going Deeper With Directly-Trained Larger Spiking Neural Networks

ReplaceNet: real-time replacement of a biological neural circuit with a hardware-assisted spiking neural network