Abstract:Transformers, originally devised for natural language processing (NLP), have also produced significant successes in computer vision (CV). Due to their strong expression power, researchers are investigating ways to deploy transformers for reinforcement learning (RL), and transformer-based models have manifested their potential in representative RL benchmarks. In this paper, we collect and dissect recent advances concerning the transformation of RL with transformers (transformer-based RL (TRL)) to explore the development trajectory and future trends of this field. We group the existing developments into two categories: architecture enhancements and trajectory optimizations, and examine the main applications of TRL in robotic manipulation, text-based games (TBGs), navigation, and autonomous driving. Architecture enhancement methods consider how to apply the powerful transformer structure to RL problems under the traditional RL framework, facilitating more precise modeling of agents and environments compared to traditional deep RL techniques. However, these methods are still limited by the inherent defects of traditional RL algorithms, such as bootstrapping and the "deadly triad". Trajectory optimization methods treat RL problems as sequence modeling problems and train a joint state-action model over entire trajectories under the behavior cloning framework; such approaches are able to extract policies from static datasets and fully use the long-sequence modeling capabilities of transformers. Given these advancements, the limitations and challenges in TRL are reviewed and proposals regarding future research directions are discussed. We hope that this survey can provide a detailed introduction to TRL and motivate future research in this rapidly developing field.

Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

Multi-Game Decision Transformers

Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models

AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers

Online Decision MetaMorphFormer: A Casual Transformer-Based Reinforcement Learning Framework of Universal Embodied Intelligence

On Transforming Reinforcement Learning by Transformer: The Development Trajectory

RT-1: Robotics Transformer for Real-World Control at Scale

GiT: Towards Generalist Vision Transformer through Universal Language Interface

On Transforming Reinforcement Learning With Transformers: The Development Trajectory

Transformer in Transformer As Backbone for Deep Reinforcement Learning

UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers

Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement

What comes after transformers? -- A selective survey connecting ideas in deep learning

Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions

A Survey on Transformers in Reinforcement Learning

Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAX

Meta-Transformer: A Unified Framework for Multimodal Learning

Multi-Agent Reinforcement Learning is a Sequence Modeling Problem

Solving Multi-Goal Robotic Tasks with Decision Transformer

AutoTaskFormer: Searching Vision Transformers for Multi-task Learning