Abstract:Understanding the intentions and beliefs of others, a phenomenon known as "theory of mind", is a crucial element in social behavior. These beliefs and perceptions are inherently subjective and latent, making them often unobservable for investigation. Social interactions further complicate the matter, as multiple agents can engage in recursive reasoning about each other's strategies with increasing levels of cognitive hierarchy. While previous research has shown promise in understanding a single agent's belief of values through inverse reinforcement learning, extending this to model interactions among multiple agents remains an open challenge due to the computational complexity. In this work, we adopted a probabilistic recursive modeling of cognitive levels and joint value decomposition to achieve efficient multi-agent inverse reinforcement learning (MAIRL). We validated our method using simulations of a cooperative foraging task. Our algorithm revealed both the ground truth goal-directed value function and agents' beliefs about their counter-parts' strategies. When applied to human behavior in a cooperative hallway task, our method identified meaningful goal maps that evolved with task proficiency and an interaction map that is related to key states in the task without accessing to the task rules. Similarly, in a non-cooperative task performed by monkeys, we identified mutual predictions that correlated with the animals' social hierarchy, highlighting the behavioral relevance of the latent beliefs we uncovered. Together, our findings demonstrate that MAIRL offers a new framework for uncovering human or animal beliefs in social behavior, thereby illuminating previously opaque aspects of social cognition.

Inverse Reinforcement Learning as the Algorithmic Basis for Theory of Mind: Current Methods and Open Problems

Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learning

A Review Of Inverse Reinforcement Learning Theory And Recent Advances

Towards Theoretical Understanding of Inverse Reinforcement Learning

On computational models of theory of mind and the imitative reinforcement learning in spiking neural networks

Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review

Theory of Mind From Observation in Cognitive Models and Humans

Multiagent Inverse Reinforcement Learning via Theory of Mind Reasoning

Mind the gap: Challenges of deep learning approaches to Theory of Mind

A Survey of Inverse Reinforcement Learning Techniques.

Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards

Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning

On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models

Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data

Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience

Interpretable Reinforcement Learning Inspired by Piaget's Theory of Cognitive Development

Reinforcement Learning and Inverse Reinforcement Learning with System 1 and System 2

A Brain-Inspired Model of Theory of Mind

Unveiling the latent dynamics in social cognition with multi-agent inverse reinforcement learning

An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey

Modeling Theory of Mind in Multi-Agent Games Using Adaptive Feedback Control