Abstract:Effectively learning from sequential data is a longstanding goal of Artificial Intelligence, especially in the case of long sequences. From the dawn of Machine Learning, several researchers engaged in the search of algorithms and architectures capable of processing sequences of patterns, retaining information about the past inputs while still leveraging the upcoming data, without losing precious long-term dependencies and correlations. While such an ultimate goal is inspired by the human hallmark of continuous real-time processing of sensory information, several solutions simplified the learning paradigm by artificially limiting the processed context or dealing with sequences of limited length, given in advance. These solutions were further emphasized by the large ubiquity of Transformers, that have initially shaded the role of Recurrent Neural Nets. However, recurrent networks are facing a strong recent revival due to the growing popularity of (deep) State-Space models and novel instances of large-context Transformers, which are both based on recurrent computations to go beyond several limits of currently ubiquitous technologies. In fact, the fast development of Large Language Models enhanced the interest in efficient solutions to process data over time. This survey provides an in-depth summary of the latest approaches that are based on recurrent models for sequential data processing. A complete taxonomy over the latest trends in architectural and algorithmic solutions is reported and discussed, guiding researchers in this appealing research field. The emerging picture suggests that there is room for thinking of novel routes, constituted by learning algorithms which depart from the standard Backpropagation Through Time, towards a more realistic scenario where patterns are effectively processed online, leveraging local-forward computations, opening to further research on this topic.

Investigating Recurrent Transformers with Dynamic Halt

Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems

Separations in the Representational Capabilities of Transformers and Recurrent Architectures

On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era

Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer

On the Design Space Between Transformers and Recursive Neural Nets

Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU

Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning

TGTOD: A Global Temporal Graph Transformer for Outlier Detection at Scale

Dynamic Context Adaptation and Information Flow Control in Transformers: Introducing the Evaluator Adjuster Unit and Gated Residual Connections

Rethinking Decision Transformer via Hierarchical Reinforcement Learning

Multiresolution Transformer Networks: Recurrence is Not Essential for Modeling Hierarchical Structure

State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

RWKV: Reinventing RNNs for the Transformer Era

Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics

Rethinking Transformers in Solving POMDPs

On Limitation of Transformer for Learning HMMs

Representational Strengths and Limitations of Transformers

Contrastive Bidirectional Transformer for Temporal Representation Learning

Jump to Conclusions: Short-Cutting Transformers With Linear Transformations