Abstract:In recent years the field of neuromorphic low-power systems that consume orders of magnitude less power gained significant momentum. However, their wider use is still hindered by the lack of algorithms that can harness the strengths of such architectures. While neuromorphic adaptations of representation learning algorithms are now emerging, efficient processing of temporal sequences or variable length-inputs remain difficult. Recurrent neural networks (RNN) are widely used in machine learning to solve a variety of sequence learning tasks. In this work we present a train-and-constrain methodology that enables the mapping of machine learned (Elman) RNNs on a substrate of spiking neurons, while being compatible with the capabilities of current and near-future neuromorphic systems. This "train-and-constrain" method consists of first training RNNs using backpropagation through time, then discretizing the weights and finally converting them to spiking RNNs by matching the responses of artificial neurons with those of the spiking neurons. We demonstrate our approach by mapping a natural language processing task (question classification), where we demonstrate the entire mapping process of the recurrent layer of the network on IBM's Neurosynaptic System "TrueNorth", a spike-based digital neuromorphic hardware architecture. TrueNorth imposes specific constraints on connectivity, neural and synaptic parameters. To satisfy these constraints, it was necessary to discretize the synaptic weights and neural activities to 16 levels, and to limit fan-in to 64 inputs. We find that short synaptic delays are sufficient to implement the dynamical (temporal) aspect of the RNN in the question classification task. The hardware-constrained model achieved 74% accuracy in question classification while using less than 0.025% of the cores on one TrueNorth chip, resulting in an estimated power consumption of ~17 uW.

Distributed Representations Enable Robust Multi-Timescale Symbolic Computation in Neuromorphic Hardware

Spiking Neurons with Neural Dynamics Implemented Using Stochastic Memristors

Computing of temporal information in spiking neural networks with ReRAM synapses

Accurate Mapping of RNNs on Neuromorphic Hardware with Adaptive Spiking Neurons

Dendritic Computation through Exploiting Resistive Memory as both Delays and Weights

Organizing Sequential Memory in a Neuromorphic Device Using Dynamic Neural Fields

Compiling Spiking Neural Networks to Neuromorphic Hardware

Neuromorphic analog circuits for robust on-chip always-on learning in spiking neural networks

Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware

In-situ Learning in Hardware Compatible Multi-layer Memristive Spiking Neural Network

Hardware Demonstration of SRDP Neuromorphic Computing with Online Unsupervised Learning Based on Memristor Synapses

A 45nm CMOS Neuromorphic Chip with a Scalable Architecture for Learning in Networks of Spiking Neurons

Scalable NoC-based Neuromorphic Hardware Learning and Inference

Stable recurrent dynamics in heterogeneous neuromorphic computing systems using excitatory and inhibitory plasticity

Homogeneous Spiking Neuromorphic System for Real-World Pattern Recognition

Spiking representation learning for associative memories

MT-Spike: A Multilayer Time-based Spiking Neuromorphic Architecture with Temporal Error Backpropagation

Stochastic dendrites enable online learning in mixed-signal neuromorphic processing systems

Reconfigurable Neural Synaptic Plasticity-Based Stochastic Deep Neural Network Computing

Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics

Demonstrating Hybrid Learning in a Flexible Neuromorphic Hardware System