Abstract:Recurrent neural networks (RNNs) are complex dynamical systems, capable of ongoing activity without any driving input. The long-term behavior of free-running RNNs, described by periodic, chaotic and fixed point attractors, is controlled by the statistics of the neural connection weights, such as the density d of non-zero connections, or the balance b between excitatory and inhibitory connections. However, for information processing purposes, RNNs need to receive external input signals, and it is not clear which of the dynamical regimes is optimal for this information import. We use both the average correlations C and the mutual information I between the momentary input vector and the next system state vector as quantitative measures of information import and analyze their dependence on the balance and density of the network. Remarkably, both resulting phase diagrams C ( b, d ) and I ( b, d ) are highly consistent, pointing to a link between the dynamical systems and the information-processing approach to complex systems. Information import is maximal not at the “edge of chaos,” which is optimally suited for computation, but surprisingly in the low-density chaotic regime and at the border between the chaotic and fixed point regime. Moreover, we find a completely new type of resonance phenomenon, which we call “Import Resonance” (IR), where the information import shows a maximum, i.e., a peak-like dependence on the coupling strength between the RNN and its external input. IR complements previously found Recurrence Resonance (RR), where correlation and mutual information of successive system states peak for a certain amplitude of noise added to the system. Both IR and RR can be exploited to optimize information processing in artificial neural networks and might also play a crucial role in biological neural systems.

Use of recurrent infomax to improve the memory capability of input-driven recurrent neural networks

Recurrent infomax generates cell assemblies, avalanches, and simple cell-like selectivity

Short term memory by transient oscillatory dynamics in recurrent neural networks

Memory and Information Processing in Recurrent Neural Networks

A Novel Framework for Recurrent Neural Networks with Enhancing Information Processing and Transmission between Units

Memory Capacity of Recurrent Neural Networks with Matrix Representation

Short-term Sequence Memory: Compressive Effects of Recurrent Network Dynamics

On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis

On the Long-Term Memory of Deep Recurrent Networks

Reversible Recurrent Neural Networks

Slow manifolds in recurrent networks encode working memory efficiently and robustly

Dual-module Inference for Efficient Recurrent Neural Networks

Recurrent Neural Networks with Finite Memory Length

Nonrecurrent Neural Structure for Long-Term Dependence.

Improving Learning Efficiency of Recurrent Neural Network Through Adjusting Weights of All Layers in a Biologically-Inspired Framework.

Hebbian and Gradient-based Plasticity Enables Robust Memory and Rapid Learning in RNNs

Dynamics and Information Import in Recurrent Neural Networks

Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks

Exploring Flip Flop memories and beyond: training Recurrent Neural Networks with key insights

Relational recurrent neural networks

Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks