Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models

Logan Cross,Violet Xiang,Agam Bhatia,Daniel LK Yamins,Nick Haber
2024-07-10
Abstract:Multi-agent reinforcement learning (MARL) methods struggle with the non-stationarity of multi-agent systems and fail to adaptively learn online when tested with novel agents. Here, we leverage large language models (LLMs) to create an autonomous agent that can handle these challenges. Our agent, Hypothetical Minds, consists of a cognitively-inspired architecture, featuring modular components for perception, memory, and hierarchical planning over two levels of abstraction. We introduce the Theory of Mind module that scaffolds the high-level planning process by generating hypotheses about other agents' strategies in natural language. It then evaluates and iteratively refines these hypotheses by reinforcing hypotheses that make correct predictions about the other agents' behavior. Hypothetical Minds significantly improves performance over previous LLM-agent and RL baselines on a range of competitive, mixed motive, and collaborative domains in the Melting Pot benchmark, including both dyadic and population-based environments. Additionally, comparisons against LLM-agent baselines and ablations reveal the importance of hypothesis evaluation and refinement for succeeding on complex scenarios.
Artificial Intelligence
What problem does this paper attempt to address?
The paper attempts to address the problem of how to create autonomous agents capable of handling non-stationarity challenges in multi-agent environments using large language models (LLMs). Specifically, multi-agent reinforcement learning (MARL) methods perform poorly when faced with non-stationary systems and cannot adaptively learn online when encountering new agents during testing. To solve these issues, the paper proposes a model called "Hypothetical Minds." This model features a cognitively inspired architecture, including modules for perception, memory, and abstract-level planning, and introduces a "Theory of Mind" (ToM) module. This ToM module enhances the high-level planning process by generating natural language hypotheses about other agents' strategies and iteratively improving these hypotheses through evaluation. Moreover, the model significantly outperforms previous LLM agents and reinforcement learning baselines in various competitive, mixed-motive, and collaborative domains in the Melting Pot benchmark, demonstrating superior performance and generalization capabilities across a wide range of evaluation scenarios.