Abstract: Large sequence model (SM) such as GPT series and BERT has displayed outstanding performance and generalization capabilities on vision, language, and recently reinforcement learning tasks. A natural follow-up question is how to abstract multi-agent decision making into an SM problem and benefit from the prosperous development of SMs. In this paper, we introduce a novel architecture named Multi-Agent Transformer (MAT) that effectively casts cooperative multi-agent reinforcement learning (MARL) into SM problems wherein the task is to map agents' observation sequence to agents' optimal action sequence. Our goal is to build the bridge between MARL and SMs so that the modeling power of modern sequence models can be unleashed for MARL. Central to our MAT is an encoder-decoder architecture which leverages the multi-agent advantage decomposition theorem to transform the joint policy search problem into a sequential decision making process; this renders only linear time complexity for multi-agent problems and, most importantly, endows MAT with monotonic performance improvement guarantee. Unlike prior arts such as Decision Transformer fit only pre-collected offline data, MAT is trained by online trials and errors from the environment in an on-policy fashion. To validate MAT, we conduct extensive experiments on StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football benchmarks. Results demonstrate that MAT achieves superior performance and data efficiency compared to strong baselines including MAPPO and HAPPO. Furthermore, we demonstrate that MAT is an excellent few-short learner on unseen tasks regardless of changes in the number of agents. See our project page at https://sites.google.com/view/multi-agent-transformer.

Models As Agents: Optimizing Multi-Step Predictions of Interactive Local Models in Model-Based Multi-Agent Reinforcement Learning

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

Prediction Learning In Multi Agent Systems

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts

Scalable Model-based Policy Optimization for Decentralized Networked Systems

Model-Based Reinforcement Learning via Meta-Policy Optimization

Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning

Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation

APC: Predict Global Representation from Local Observation in Multi-Agent Reinforcement Learning

Characterizing and Optimizing the End-to-End Performance of Multi-Agent Reinforcement Learning Systems

Multi-Agent Reinforcement Learning is a Sequence Modeling Problem

Bidirectional Model-based Policy Optimization

Plan to Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learning.

Cooperative Multi-Agent Reinforcement Learning with Partial Observations

LMRL: a Multi-Agent Reinforcement Learning Model and Algorithm

Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects

Modeling opponent learning in multiagent repeated games

Combating the Compounding-Error Problem with a Multi-step Model

Off-Beat Multi-Agent Reinforcement Learning