Abstract:Spiking Neural Networks (SNNs) have emerged as a hardware efficient architecture for classification tasks. The challenge of spike-based encoding has been the lack of a universal training mechanism performed entirely using spikes. There have been several attempts to adopt the powerful backpropagation (BP) technique used in non-spiking artificial neural networks (ANN): (1) SNNs can be trained by externally computed numerical gradients. (2) A major advancement towards native spike-based learning has been the use of approximate Backpropagation using spike-time dependent plasticity (STDP) with phased forward/backward passes. However, the transfer of information between such phases for gradient and weight update calculation necessitates external memory and computational access. This is a challenge for standard neuromorphic hardware implementations. In this paper, we propose a stochastic SNN based Back-Prop (SSNN-BP) algorithm that utilizes a composite neuron to simultaneously compute the forward pass activations and backward pass gradients explicitly with spikes. Although signed gradient values are a challenge for spike-based representation, we tackle this by splitting the gradient signal into positive and negative streams. The composite neuron encodes information in the form of stochastic spike-trains and converts Backpropagation weight updates into temporally and spatially local spike coincidence updates compatible with hardware-friendly Resistive Processing Units (RPUs). Furthermore, we characterize the quantization effect of discrete spike-based weight update to show that our method approaches BP ANN baseline with sufficiently long spike-trains. Finally, we show that the well-performing softmax cross-entropy loss function can be implemented through inhibitory lateral connections enforcing a Winner Take All (WTA) rule. Our SNN with a 2-layer network shows excellent generalization through comparable performance to ANNs with equivalent architecture and regularization parameters on static image datasets like MNIST, Fashion-MNIST, Extended MNIST, and temporally encoded image datasets like Neuromorphic MNIST datasets. Thus, SSNN-BP enables BP compatible with purely spike-based neuromorphic hardware.

Enhancing performance of the backpropagation algorithm via sparse response regularization.

Backpropagation with Sparsity Regularization for Spiking Neural Network Learning.

ssProp: Energy-Efficient Training for Convolutional Neural Networks with Scheduled Sparse Back Propagation

SRS-DNN: a Deep Neural Network with Strengthening Response Sparsity

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Back Propagation Based on Selective Attention for Fast Convergence of Training Neural Network

BP-SRM: A directly training algorithm for spiking neural network constructed by spike response model

Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks

Optimal parameters selection of back propagation algorithm in the feedforward neural network

Training Spiking Neural Networks with Accumulated Spiking Flow.

Memorized Sparse Backpropagation.

Efficient Neural Network Training Via Forward and Backward Propagation Sparsification

A temporally and spatially local spike-based backpropagation algorithm to enable training in hardware

An Algorithm for Fast Convergence of Back Propagation by Enlarging Error

Spatiotemporal Backpropagation based on Channel Reward for Training High-Precision Spiking Neural Network

Fast Sparse Deep Neural Networks: Theory and Performance Analysis

Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks

Intralayer-Connected Spiking Neural Network with Hybrid Training Using Backpropagation and Probabilistic Spike-Timing Dependent Plasticity

Tuning Convolutional Spiking Neural Network with Biologically-plausible Reward Propagation

FastLRNR and Sparse Physics Informed Backpropagation

Convergence Analysis of Novel Fractional-Order Backpropagation Neural Networks With Regularization Terms