Abstract:Spiking Neural Networks (SNNs) have emerged as a hardware efficient architecture for classification tasks. The challenge of spike-based encoding has been the lack of a universal training mechanism performed entirely using spikes. There have been several attempts to adopt the powerful backpropagation (BP) technique used in non-spiking artificial neural networks (ANN): (1) SNNs can be trained by externally computed numerical gradients. (2) A major advancement towards native spike-based learning has been the use of approximate Backpropagation using spike-time dependent plasticity (STDP) with phased forward/backward passes. However, the transfer of information between such phases for gradient and weight update calculation necessitates external memory and computational access. This is a challenge for standard neuromorphic hardware implementations. In this paper, we propose a stochastic SNN based Back-Prop (SSNN-BP) algorithm that utilizes a composite neuron to simultaneously compute the forward pass activations and backward pass gradients explicitly with spikes. Although signed gradient values are a challenge for spike-based representation, we tackle this by splitting the gradient signal into positive and negative streams. The composite neuron encodes information in the form of stochastic spike-trains and converts Backpropagation weight updates into temporally and spatially local spike coincidence updates compatible with hardware-friendly Resistive Processing Units (RPUs). Furthermore, we characterize the quantization effect of discrete spike-based weight update to show that our method approaches BP ANN baseline with sufficiently long spike-trains. Finally, we show that the well-performing softmax cross-entropy loss function can be implemented through inhibitory lateral connections enforcing a Winner Take All (WTA) rule. Our SNN with a 2-layer network shows excellent generalization through comparable performance to ANNs with equivalent architecture and regularization parameters on static image datasets like MNIST, Fashion-MNIST, Extended MNIST, and temporally encoded image datasets like Neuromorphic MNIST datasets. Thus, SSNN-BP enables BP compatible with purely spike-based neuromorphic hardware.

Pattern Recognition with Spiking Neural Networks

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

Pattern Recognition Computation in a Spiking Neural Network with Temporal Encoding and Learning

Event-driven Spiking Neural Networks with Spike-Based Learning

A Spike-Timing Based Integrated Model for Pattern Recognition

A Supervised Multi-Spike Learning Algorithm for Spiking Neural Networks

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks

Supervised learning of spatial features with STDP and homeostasis using Spiking Neural Networks on SpiNNaker

Spiking Neural Networks -- Part II: Detecting Spatio-Temporal Patterns

A temporally and spatially local spike-based backpropagation algorithm to enable training in hardware

Backpropagation-Based Learning Techniques for Deep Spiking Neural Networks: A Survey

A Parallel Spiking Neural Network Based on Adaptive Lateral Inhibition Mechanism for Objective Recognition

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

A Short Survey of the Development and Applications of Spiking Neural Networks of High Biological Plausibility

Research Progress of spiking neural network in image classification: a review

Spatial-Temporal Search for Spiking Neural Networks

A Review of Recent Advances and Application for Spiking Neural Networks

Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

An Efficient Software-Hardware Design Framework for Spiking Neural Network Systems

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization