Unveiling the latent dynamics in social cognition with multi-agent inverse reinforcement learning

Yusi Chen,Angela Radulescu,Herbert Zheng Wu
DOI: https://doi.org/10.1101/2024.10.09.617461
2024-10-12
Abstract:Understanding the intentions and beliefs of others, a phenomenon known as "theory of mind", is a crucial element in social behavior. These beliefs and perceptions are inherently subjective and latent, making them often unobservable for investigation. Social interactions further complicate the matter, as multiple agents can engage in recursive reasoning about each other's strategies with increasing levels of cognitive hierarchy. While previous research has shown promise in understanding a single agent's belief of values through inverse reinforcement learning, extending this to model interactions among multiple agents remains an open challenge due to the computational complexity. In this work, we adopted a probabilistic recursive modeling of cognitive levels and joint value decomposition to achieve efficient multi-agent inverse reinforcement learning (MAIRL). We validated our method using simulations of a cooperative foraging task. Our algorithm revealed both the ground truth goal-directed value function and agents' beliefs about their counter-parts' strategies. When applied to human behavior in a cooperative hallway task, our method identified meaningful goal maps that evolved with task proficiency and an interaction map that is related to key states in the task without accessing to the task rules. Similarly, in a non-cooperative task performed by monkeys, we identified mutual predictions that correlated with the animals' social hierarchy, highlighting the behavioral relevance of the latent beliefs we uncovered. Together, our findings demonstrate that MAIRL offers a new framework for uncovering human or animal beliefs in social behavior, thereby illuminating previously opaque aspects of social cognition.
Animal Behavior and Cognition
What problem does this paper attempt to address?
The paper attempts to address the problem of understanding others' intentions and beliefs in social cognition (i.e., "theory of mind"). These beliefs and perceptions are inherently subjective and latent, making them often unobservable in research. Social interactions further complicate this issue as multiple agents can engage in recursive reasoning about each other's strategies, forming different levels of cognitive hierarchies. Although previous studies have shown the potential of understanding single-agent value beliefs through Inverse Reinforcement Learning (IRL), extending this to model interactions among multiple agents remains an open challenge, primarily due to high computational complexity. To address this problem, the authors adopt a probabilistic recursive modeling approach to handle cognitive hierarchies and achieve efficient Multi-Agent Inverse Reinforcement Learning (MAIRL) through joint value decomposition. This approach not only reveals goal-oriented value functions but also uncovers agents' beliefs about their peers' strategies. The authors validate their method through simulations of cooperative foraging tasks and apply it to human cooperative corridor tasks and non-cooperative tasks with monkeys, successfully identifying meaningful goal maps and interaction maps related to task-critical states, as well as mutual predictions related to animals' social hierarchies. These findings suggest that MAIRL provides a new framework for uncovering beliefs in human or animal social behavior, thereby elucidating previously opaque aspects of social cognition.