Abstract:Recent breakthroughs in AI for multi-agent games like Go, Poker, and Dota, have seen great strides in recent years. Yet none of these games address the real-life challenge of cooperation in the presence of unknown and uncertain teammates. This challenge is a key game mechanism in hidden role games. Here we develop the DeepRole algorithm, a multi-agent reinforcement learning agent that we test on The Resistance: Avalon, the most popular hidden role game. DeepRole combines counterfactual regret minimization (CFR) with deep value networks trained through self-play. Our algorithm integrates deductive reasoning into vector-form CFR to reason about joint beliefs and deduce partially observable actions. We augment deep value networks with constraints that yield interpretable representations of win probabilities. These innovations enable DeepRole to scale to the full Avalon game. Empirical game-theoretic methods show that DeepRole outperforms other hand-crafted and learned agents in five-player Avalon. DeepRole played with and against human players on the web in hybrid human-agent teams. We find that DeepRole outperforms human players as both a cooperator and a competitor.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to cooperate effectively in multi - agent games, especially in collaborative games with hidden roles, when the identities of teammates are unknown and uncertain. Specifically, the paper focuses on how to enable agents to learn to identify which players are allies (objects that can be cooperated with) and which players are opponents (objects that need to be guarded against). This challenge also exists in real life. For example, in scenarios such as social interactions and business negotiations, it is necessary to quickly and accurately judge the intentions of others. ### Background and Problem Description of the Paper In recent years, AI has made remarkable progress in multi - agent games (such as Go, Poker, Dota, etc.), but in these games, the objects of cooperation are usually known and certain. However, in the real world, especially in some complex social and economic environments, the identities of cooperation objects are often uncertain, which increases the difficulty of cooperation. For example, in role - hiding games such as "Werewolf" and "The Resistance: Avalon", players need to infer the roles of others based on limited information and decide whether to cooperate with them. ### Specific Problems 1. **How to conduct effective reasoning and decision - making under incomplete information**: In role - hiding games, there is uncertainty between players' behaviors and roles, and agents need to infer their real identities by observing the behaviors of other players. 2. **How to handle partially observable actions**: In some games, some actions can only be seen by specific players, and agents need to reason based on indirect information. 3. **How to achieve efficient strategy learning in a multi - agent environment**: Traditional reinforcement learning methods face challenges when dealing with multi - agent environments, especially when cooperation and competition coexist. ### Solutions To address the above challenges, the paper proposes an algorithm named DeepRole. DeepRole combines Counterfactual Regret Minimization (CFR) and a deep value network and is trained through self - play. Specific innovation points include: - **Introducing deductive reasoning into CFR**: Enhance the CFR algorithm through deductive reasoning, enabling it to handle joint beliefs and partially observable actions. - **Enhancing the interpretability of the deep value network**: Impose constraints on the value network to make its output an interpretable win - rate representation. - **Handling large - scale state spaces**: Limit the search depth of the game tree through a neural network, enabling the algorithm to operate efficiently in complex game environments. ### Experimental Verification The paper tested the performance of DeepRole in the five - person version of "The Resistance: Avalon" game. The experimental results show that DeepRole not only performs excellently in comparison with hand - written rules or learning - type agents, but also performs well in mixed teams with human players. Whether as a teammate or an opponent, DeepRole can outperform human players. ### Conclusion The success of DeepRole shows that in role - hiding games, agents can successfully identify cooperation objects and achieve victory in complex environments through effective reasoning and learning mechanisms. This result provides new ideas and methods for future research on multi - agent systems. --- I hope this summary can help you understand the core problems of this paper and their solutions. If you have more questions or need further explanations, please feel free to let me know!

Finding Friend and Foe in Multi-Agent Games

Towards Cooperation in Sequential Prisoner's Dilemmas: a Deep Multiagent Reinforcement Learning Approach

RODE: Learning Roles to Decompose Multi-Agent Tasks

Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions

Effective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principles

A Unified Perspective on Deep Equilibrium Finding

Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria

Classifying ambiguous identities in hidden-role Stochastic games with multi-agent reinforcement learning

Resolving Implicit Coordination in Multi-Agent Deep Reinforcement Learning with Deep Q-Networks & Game Theory

Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning

Hierarchical Deep Reinforcement Learning Agent with Counter Self-play on Competitive Games

Neural Auto-Curricula

Cooperative multi-agent game based on reinforcement learning

From Text to Tactic: Evaluating LLMs Playing the Game of Avalon

Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination

Neural Auto-Curricula in Two-Player Zero-Sum Games.

Deep reinforcement learning algorithm based on multi-agent parallelism and its application in game environment

Algorithms in Multi-Agent Systems: A Holistic Perspective from Reinforcement Learning and Game Theory

Attention-Guided Contrastive Role Representations for Multi-Agent Reinforcement Learning

Mastering the game of Stratego with model-free multiagent reinforcement learning