Abstract:Spiking Neural Networks (SNNs) have emerged as a hardware efficient architecture for classification tasks. The challenge of spike-based encoding has been the lack of a universal training mechanism performed entirely using spikes. There have been several attempts to adopt the powerful backpropagation (BP) technique used in non-spiking artificial neural networks (ANN): (1) SNNs can be trained by externally computed numerical gradients. (2) A major advancement towards native spike-based learning has been the use of approximate Backpropagation using spike-time dependent plasticity (STDP) with phased forward/backward passes. However, the transfer of information between such phases for gradient and weight update calculation necessitates external memory and computational access. This is a challenge for standard neuromorphic hardware implementations. In this paper, we propose a stochastic SNN based Back-Prop (SSNN-BP) algorithm that utilizes a composite neuron to simultaneously compute the forward pass activations and backward pass gradients explicitly with spikes. Although signed gradient values are a challenge for spike-based representation, we tackle this by splitting the gradient signal into positive and negative streams. The composite neuron encodes information in the form of stochastic spike-trains and converts Backpropagation weight updates into temporally and spatially local spike coincidence updates compatible with hardware-friendly Resistive Processing Units (RPUs). Furthermore, we characterize the quantization effect of discrete spike-based weight update to show that our method approaches BP ANN baseline with sufficiently long spike-trains. Finally, we show that the well-performing softmax cross-entropy loss function can be implemented through inhibitory lateral connections enforcing a Winner Take All (WTA) rule. Our SNN with a 2-layer network shows excellent generalization through comparable performance to ANNs with equivalent architecture and regularization parameters on static image datasets like MNIST, Fashion-MNIST, Extended MNIST, and temporally encoded image datasets like Neuromorphic MNIST datasets. Thus, SSNN-BP enables BP compatible with purely spike-based neuromorphic hardware.

Hardware implementation of single-bit nonlinear BP neural network based on sigma-delta modulation

ASIC Implementation of a Nonlinear Dynamical Model for Hippocampal Prosthesis

Darwin:a Neuromorphic Hardware Co-Processor Based on Spiking Neural Networks

FPGA implementation of hardware processing modules as coprocessors in brain-machine interfaces.

Hardware implementation of an on-chip BP learning neural network with programmable neuron characteristics and learning rate adaptation

Analogue Circuit Realization of a Programmable Sigmoidal Function and Its Derivative for On-Chip BP Learning

A Sparsity-Adapted Hardware Implementation of SNN for Cortical Spike Trains Decoding

A programmable on-chip BP learning neural network with enhanced neuron characteristics

A Fast Spiking Neural Network Accelerator based on BP-STDP Algorithm and Weighted Neuron Model

A temporally and spatially local spike-based backpropagation algorithm to enable training in hardware

Field-programmable gate array implementation of a probabilistic neural network for motor cortical decoding in rats.

A Single Layer Architecture to FPGA Implementation of BP Artificial Neural Network

Hardware implementation of spiking neural networks on FPGA

Hardware implementation of radial-basis neural networks with Gaussian activation functions on FPGA

FP-BNN: Binarized neural network on FPGA

Hardware Realization of Building Blocks for Artificial Neural Networks

A FPGA-based Hardware Accelerator for Bayesian Confidence Propagation Neural Network

A New Hardware Implementation Approach Of Bnns Based On Nonlinear 2t2r Synaptic Cell

Hardware Implementation Of Rram Based Binarized Neural Networks

AHEAD: Automatic Holistic Energy-Aware Design Methodology for MLP Neural Network Hardware Generation in Proactive BMI Edge Devices

Base-Reconfigurable Segmented Logarithmic Quantization and Hardware Design for Deep Neural Networks