Abstract:In recent years the field of neuromorphic low-power systems that consume orders of magnitude less power gained significant momentum. However, their wider use is still hindered by the lack of algorithms that can harness the strengths of such architectures. While neuromorphic adaptations of representation learning algorithms are now emerging, efficient processing of temporal sequences or variable length-inputs remain difficult. Recurrent neural networks (RNN) are widely used in machine learning to solve a variety of sequence learning tasks. In this work we present a train-and-constrain methodology that enables the mapping of machine learned (Elman) RNNs on a substrate of spiking neurons, while being compatible with the capabilities of current and near-future neuromorphic systems. This "train-and-constrain" method consists of first training RNNs using backpropagation through time, then discretizing the weights and finally converting them to spiking RNNs by matching the responses of artificial neurons with those of the spiking neurons. We demonstrate our approach by mapping a natural language processing task (question classification), where we demonstrate the entire mapping process of the recurrent layer of the network on IBM's Neurosynaptic System "TrueNorth", a spike-based digital neuromorphic hardware architecture. TrueNorth imposes specific constraints on connectivity, neural and synaptic parameters. To satisfy these constraints, it was necessary to discretize the synaptic weights and neural activities to 16 levels, and to limit fan-in to 64 inputs. We find that short synaptic delays are sufficient to implement the dynamical (temporal) aspect of the RNN in the question classification task. The hardware-constrained model achieved 74% accuracy in question classification while using less than 0.025% of the cores on one TrueNorth chip, resulting in an estimated power consumption of ~17 uW.

Evolving Connectivity for Recurrent Spiking Neural Networks

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

Brain-inspired Evolutionary Architectures for Spiking Neural Networks

Evolving Efficient Genetic Encoding for Deep Spiking Neural Networks

ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks

Composing Recurrent Spiking Neural Networks using Locally-Recurrent Motifs and Risk-Mitigating Architectural Optimization

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Multi-scale Evolutionary Neural Architecture Search for Deep Spiking Neural Networks

Emergence of brain-inspired small-world spiking neural network through neuroevolution

Neuroevolution of a Recurrent Neural Network for Spatial and Working Memory in a Simulated Robotic Environment

Expanding memory in recurrent spiking networks

Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning

Advancing Spiking Neural Networks Toward Deep Residual Learning

Neurogenesis Dynamics-inspired Spiking Neural Network Training Acceleration

Neuroevolution of Recurrent Architectures on Control Tasks

Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware

Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning

Skip-Connected Self-Recurrent Spiking Neural Networks with Joint Intrinsic Parameter and Synaptic Weight Training

Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks

Evolving unipolar memristor spiking neural networks