Abstract:Abstract As an important class of spiking neural networks (SNNs), recurrent spiking neural networks (RSNNs) possess great computational power and have been widely used for pro cessing sequential data like audio and text. However, most RSNNs suffer from two problems. First, due to the lack of architectural guidance, random recurrent connectivity is often adopted, which does not guarantee good performance. Second, training of RSNNs is in general challenging, bottlenecking achievable model accuracy. To address these problems, we propose a new type of RSNN, skip-connected self-recurrent SNNs (ScSr-SNNs). Recurrence in ScSr-SNNs is introduced by adding self-recurrent connections to spiking neurons. The SNNs with self-recurrent connections can realize recurrent behaviors similar to those of more complex RSNNs, while the error gradients can be more straightforwardly calculated due to the mostly feedforward nature of the network. The network dynamics is enriched by skip connections between nonadjacent layers. Moreover, we propose a new backpropagation (BP) method, backpropagated intrinsic plasticity (BIP), to boost the performance of ScSr-SNNs further by training intrinsic model parameters. Unlike standard intrinsic plasticity rules that adjust the neuron's intrinsic parameters according to neuronal activity, the proposed BIP method optimizes intrinsic parameters based on the backpropagated error gradient of a well-defined global loss function in addition to synaptic weight training. Based on challenging speech, neuromorphic speech, and neuromorphic image data sets, the proposed ScSr-SNNs can boost performance by up to 2.85% compared with other types of RSNNs trained by state-of-the-art BP methods.

Complex Recurrent Spectral Network

Recurrent Spectral Network (RSN): shaping the basin of attraction of a discrete map to reach automated classification

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

A Spiking Neural Network System for Robust Sequence Recognition

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Recurrent Control Nets for Deep Reinforcement Learning

Nonrecurrent Neural Structure for Long-Term Dependence.

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

Recurrent networks recognize patterns with low-dimensional oscillations

Recurrent neural networks as neuro-computational models of human speech recognition

Resurrecting Recurrent Neural Networks for Long Sequences

Skip-Connected Self-Recurrent Spiking Neural Networks with Joint Intrinsic Parameter and Synaptic Weight Training

Modeling Neural Representation of Social Recognition Information with Recurrent Neural Networks

Evolving Connectivity for Recurrent Spiking Neural Networks

Deep Complex Networks

Recurrent Stochastic Configuration Networks for Temporal Data Analytics

Deep Recurrent Stochastic Configuration Networks for Modelling Nonlinear Dynamic Systems

Expanding memory in recurrent spiking networks

Universal Approximation Property of Stochastic Configuration Networks for Time Series

Memory and Information Processing in Recurrent Neural Networks