Abstract:Sequential memory, the ability to form and accurately recall a sequence of events or stimuli in the correct order, is a fundamental prerequisite for biological and artificial intelligence as it underpins numerous cognitive functions (e.g., language comprehension, planning, episodic memory formation, etc.) However, existing methods of sequential memory suffer from catastrophic forgetting, limited capacity, slow iterative learning procedures, low-order Markov memory, and, most importantly, the inability to represent and generate multiple valid future possibilities stemming from the same context. Inspired by biologically plausible neuroscience theories of cognition, we propose \textit{Predictive Attractor Models (PAM)}, a novel sequence memory architecture with desirable generative properties. PAM is a streaming model that learns a sequence in an online, continuous manner by observing each input \textit{only once}. Additionally, we find that PAM avoids catastrophic forgetting by uniquely representing past context through lateral inhibition in cortical minicolumns, which prevents new memories from overwriting previously learned knowledge. PAM generates future predictions by sampling from a union set of predicted possibilities; this generative ability is realized through an attractor model trained alongside the predictor. We show that PAM is trained with local computations through Hebbian plasticity rules in a biologically plausible framework. Other desirable traits (e.g., noise tolerance, CPU-based learning, capacity scaling) are discussed throughout the paper. Our findings suggest that PAM represents a significant step forward in the pursuit of biologically plausible and computationally efficient sequential memory models, with broad implications for cognitive science and artificial intelligence research.

Efficient Multi-Sequence Memory with Controllable Steady-State Period and High Sequence Storage Capacity

Robust Sequence Memory in Sparsely-Connected Networks with Controllable Steady-State Period

Long Sequence Hopfield Memory

Long sequence Hopfield memory*

A hippocampus CA3 spiking neural network model for storage and retrieval of sequential memory

A Hierarchical Organized Memory Model Using Spiking Neurons

Sequential Memory: a Putative Neural and Synaptic Dynamical Mechanism.

Sequential Memory with Temporal Predictive Coding

Organizing Sequential Memory in a Neuromorphic Device Using Dynamic Neural Fields

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

A Brain-Inspired In-Memory Computing System for Neuronal Communication Via Memristive Circuits.

Temporal-Sequential Learning with Columnar-Structured Spiking Neural Networks.

Sequence memory for prediction, inference and behaviour

Spontaneous Temporal Grouping Neural Network for Long-Term Memory Modeling

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

Short-term Sequence Memory: Compressive Effects of Recurrent Network Dynamics

Minicolumn-Based Episodic Memory Model With Spiking Neurons, Dendrites and Delays

A theory of sequence indexing and working memory in recurrent neural networks

An Optimization Scheme for Segmented-Memory Neural Network

Predictive Attractor Models

A memory neural system built based on spiking neural network