Abstract:Sequential learning involves learning tasks in a sequence, and proves challenging for most neural networks. Biological neural networks regularly conquer the sequential learning challenge and are even capable of transferring knowledge both forward and backwards between tasks. Artificial neural networks often totally fail to transfer performance between tasks, and regularly suffer from degraded performance or catastrophic forgetting on previous tasks. Models of associative memory have been used to investigate the discrepancy between biological and artificial neural networks due to their biological ties and inspirations, of which the Hopfield network is perhaps the most studied model. The Dense Associative Memory, or modern Hopfield network, generalizes the Hopfield network, allowing for greater capacities and prototype learning behaviors, while still retaining the associative memory structure. We investigate the performance of the Dense Associative Memory in sequential learning problems, and benchmark various sequential learning techniques in the network. We give a substantial review of the sequential learning space with particular respect to the Hopfield network and associative memories, as well as describe the techniques we implement in detail. We also draw parallels between the classical and Dense Associative Memory in the context of sequential learning, and discuss the departures from biological inspiration that may influence the utility of the Dense Associative Memory as a tool for studying biological neural networks. We present our findings, and show that existing sequential learning methods can be applied to the Dense Associative Memory to improve sequential learning performance.

Sequential Back-Propagation

Residual Recurrent Neural Networks for Learning Sequential Representations.

On Training Recurrent Networks with Truncated Backpropagation Through Time in Speech Recognition

A Spiking Neural Network System for Robust Sequence Recognition

Rapid Feedforward Computation by Temporal Encoding and Learning with Spiking Neurons

Optoelectronic Systems Trained With Backpropagation Through Time

Complex Sequential Understanding Through the Awareness of Spatial and Temporal Concepts

Pattern Recognition Computation in a Spiking Neural Network with Temporal Encoding and Learning

Sequential Memory with Temporal Predictive Coding

Coherent noise enables probabilistic sequence replay in spiking neuronal networks

A hippocampus CA3 spiking neural network model for storage and retrieval of sequential memory

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

Error Back Propagation for Sequence Training of Context-Dependent Deep NetworkS for Conversational Speech Transcription

Biologically Plausible Sequence Learning with Spiking Neural Networks.

Learning The Sequential Temporal Information with Recurrent Neural Networks

Recurrent Network Models Of Sequence Generation And Memory

Long Short-Term Memory with Dynamic Skip Connections.

Sparse Attentive Backtracking: Temporal CreditAssignment Through Reminding

Semi-Implicit Back Propagation.

Sequential Learning in the Dense Associative Memory

Sequential Memory: a Putative Neural and Synaptic Dynamical Mechanism.