Abstract:Neural oscillations are ubiquitously observed in many brain areas. One proposed functional role of these oscillations is that they serve as an internal clock, or 'frame of reference'. Information can be encoded by the timing of neural activity relative to the phase of such oscillations. In line with this hypothesis, there have been multiple empirical observations of such phase codes in the brain. Here we ask: What kind of neural dynamics support phase coding of information with neural oscillations? We tackled this question by analyzing recurrent neural networks (RNNs) that were trained on a working memory task. The networks were given access to an external reference oscillation and tasked to produce an oscillation, such that the phase difference between the reference and output oscillation maintains the identity of transient stimuli. We found that networks converged to stable oscillatory dynamics. Reverse engineering these networks revealed that each phase-coded memory corresponds to a separate limit cycle attractor. We characterized how the stability of the attractor dynamics depends on both reference oscillation amplitude and frequency, properties that can be experimentally observed. To understand the connectivity structures that underlie these dynamics, we showed that trained networks can be described as two phase-coupled oscillators. Using this insight, we condensed our trained networks to a reduced model consisting of two functional modules: One that generates an oscillation and one that implements a coupling function between the internal oscillation and external reference. In summary, by reverse engineering the dynamics and connectivity of trained RNNs, we propose a mechanism by which neural networks can harness reference oscillations for working memory. Specifically, we propose that a phase-coding network generates autonomous oscillations which it couples to an external reference oscillation in a multi-stable fashion. Many of our actions are rhythmic—walking, breathing, digesting and more. It is not surprising that neural activity can have a strong oscillatory component. Indeed, such brain waves are common, and can even be measured using EEG from the scalp. Perhaps less obvious is the presence of such oscillations during non-rhythmic behavior—such as memory maintenance and other cognitive functions. Reports of these cognitive oscillations have accumulated over the years, and various theories were raised regarding their origin and utilization. In particular, oscillations have been proposed to serve as a clock signal that can be used for temporal-, or phase-coding of information in working memory. Here, we studied the dynamical systems underlying this kind of coding, by using trained artificial neural networks as hypothesis generators. We trained recurrent neural networks to perform a working memory task, while giving them access to a reference oscillation. We were then able to reverse engineer the learned dynamics of the networks. Our analysis revealed that phase-coded memories correspond to stable attractors in the dynamical landscape of the model. These attractors arose from the coupling of the external reference oscillation with oscillations generated internally by the network.

Short term memory by transient oscillatory dynamics in recurrent neural networks

Slow manifolds in recurrent networks encode working memory efficiently and robustly

Short-Term Memory Through Persistent Activity: Evolution of Self-Stopping and Self-Sustaining Activity in Spiking Neural Networks

Long-term memory stabilized by noise-induced rehearsal

Trained recurrent neural networks develop phase-locked limit cycles in a working memory task

Memory and Information Processing in Recurrent Neural Networks

Extending dynamic memory of spiking neuron networks

Use of recurrent infomax to improve the memory capability of input-driven recurrent neural networks

Slow manifolds within network dynamics encode working memory efficiently and robustly

Recurrent Network Models Of Sequence Generation And Memory

Short-term Sequence Memory: Compressive Effects of Recurrent Network Dynamics

Engineering recurrent neural networks from task-relevant manifolds and dynamics

An Oscillator Neural Network Retrieving Sparsely Coded Phase Patterns

Network mechanisms of working memory: the role of neuronal nonlinearities

Oscillations in an artificial neural network convert competing inputs into a temporal code

Transitions among metastable states underlie context-dependent working memories in a multiple timescale network

Resonant synchronization and information retrieve from memorized Kuramoto network

Stimulus-Driven and Spontaneous Dynamics in Excitatory-Inhibitory Recurrent Neural Networks for Sequence Representation

Neural networks with transient state dynamics

Short-term Plasticity As a Mechanism to Regulate and Retain Multistability