Abstract:To accommodate structured approaches of neural computation, we propose a class of recurrent neural networks for indexing and storing sequences of symbols or analog data vectors. These networks with randomized input weights and orthogonal recurrent weights implement coding principles previously described in vector symbolic architectures (VSA), and leverage properties of reservoir computing. In general, the storage in reservoir computing is lossy and crosstalk noise limits the retrieval accuracy and information capacity. A novel theory to optimize memory performance in such networks is presented and compared with simulation experiments. The theory describes linear readout of analog data, and readout with winner-take-all error correction of symbolic data as proposed in VSA models. We find that diverse VSA models from the literature have universal performance properties, which are superior to what previous analyses predicted. Further, we propose novel VSA models with the statistically optimal Wiener filter in the readout that exhibit much higher information capacity, in particular for storing analog data. The presented theory also applies to memory buffers, networks with gradual forgetting, which can operate on infinite data streams without memory overflow. Interestingly, we find that different forgetting mechanisms, such as attenuating recurrent weights or neural nonlinearities, produce very similar behavior if the forgetting time constants are aligned. Such models exhibit extensive capacity when their forgetting time constant is optimized for given noise conditions and network size. These results enable the design of new types of VSA models for the online processing of data streams.

Robust Sequence Memory in Sparsely-Connected Networks with Controllable Steady-State Period

Efficient Multi-Sequence Memory with Controllable Steady-State Period and High Sequence Storage Capacity

Long Sequence Hopfield Memory

Long sequence Hopfield memory*

A hippocampus CA3 spiking neural network model for storage and retrieval of sequential memory

A Hierarchical Organized Memory Model Using Spiking Neurons

How the Brain Formulates Memory: A Spatio-Temporal Model Research Frontier.

A Spiking Neural Network System for Robust Sequence Recognition

Sequential Memory: a Putative Neural and Synaptic Dynamical Mechanism.

Spiking Neuron Based Cognitive Memory Model

A Spiking Neural Network Model For Associative Memory Using Temporal Codes

Sequential Memory with Temporal Predictive Coding

Recurrent Network Models Of Sequence Generation And Memory

A Hierarchically Organized Memory Model With Temporal Population Coding

A theory of sequence indexing and working memory in recurrent neural networks

Short-term Sequence Memory: Compressive Effects of Recurrent Network Dynamics

Sequence Structure Organizes Items in Varied Latent States of Working Memory Neural Network.

Temporal-Sequential Learning with Columnar-Structured Spiking Neural Networks.

Spontaneous Temporal Grouping Neural Network for Long-Term Memory Modeling

The impact of memory on learning sequence-to-sequence tasks

A Spike-Timing Pattern Based Neural Network Model for the Study of Memory Dynamics