Abstract:Transformers, originally devised for natural language processing (NLP), have also produced significant successes in computer vision (CV). Due to their strong expression power, researchers are investigating ways to deploy transformers for reinforcement learning (RL), and transformer-based models have manifested their potential in representative RL benchmarks. In this paper, we collect and dissect recent advances concerning the transformation of RL with transformers (transformer-based RL (TRL)) to explore the development trajectory and future trends of this field. We group the existing developments into two categories: architecture enhancements and trajectory optimizations, and examine the main applications of TRL in robotic manipulation, text-based games (TBGs), navigation, and autonomous driving. Architecture enhancement methods consider how to apply the powerful transformer structure to RL problems under the traditional RL framework, facilitating more precise modeling of agents and environments compared to traditional deep RL techniques. However, these methods are still limited by the inherent defects of traditional RL algorithms, such as bootstrapping and the "deadly triad". Trajectory optimization methods treat RL problems as sequence modeling problems and train a joint state-action model over entire trajectories under the behavior cloning framework; such approaches are able to extract policies from static datasets and fully use the long-sequence modeling capabilities of transformers. Given these advancements, the limitations and challenges in TRL are reviewed and proposals regarding future research directions are discussed. We hope that this survey can provide a detailed introduction to TRL and motivate future research in this rapidly developing field.

The evolution of transformer models from unidirectional to bidirectional in Natural Language Processing

Advancements in Natural language Processing: An In-depth Review of Language Transformer Models

Overview of the Transformer-based Models for NLP Tasks

Transformer models in biomedicine

Language modeling and bidirectional coders representations: an overview of key technologies

Evolution and advancements in deep learning models for Natural Language Processing

Applications of transformer-based language models in bioinformatics: a survey

Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet

Natural language processing with transformers: a review

TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding

Enhanced Transformer Architecture for Natural Language Processing

Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision

From Turing to Transformers: A Comprehensive Review and Tutorial on the Evolution and Applications of Generative Transformer Models

Transformers in Natural Language Processing: A Comprehensive Review

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

End-to-End Transformer-Based Models in Textual-Based NLP

The Antecedents of Transformer Models

On Transforming Reinforcement Learning by Transformer: The Development Trajectory

What comes after transformers? -- A selective survey connecting ideas in deep learning

On Transforming Reinforcement Learning With Transformers: The Development Trajectory

Generative AI in the Era of Transformers: Revolutionizing Natural Language Processing with LLMs