Abstract:To learn good joint policies for multi-agent collaboration with imperfect information remains a fundamental challenge. While for two-player zero-sum games, coordinate-ascent approaches (optimizing one agent's policy at a time, e.g., self-play) work with guarantees, in multi-agent cooperative setting they often converge to sub-optimal Nash equilibrium. On the other hand, directly modeling joint policy changes in imperfect information game is nontrivial due to complicated interplay of policies (e.g., upstream updates affect downstream state reachability). In this paper, we show global changes of game values can be decomposed to policy changes localized at each information set, with a novel term named policy-change density. Based on this, we propose Joint Policy Search(JPS) that iteratively improves joint policies of collaborative agents in imperfect information games, without re-evaluating the entire game. On multi-agent collaborative tabular games, JPS is proven to never worsen performance and can improve solutions provided by unilateral approaches (e.g, CFR), outperforming algorithms designed for collaborative policy learning (e.g. BAD). Furthermore, for real-world games, JPS has an online form that naturally links with gradient updates. We test it to Contract Bridge, a 4-player imperfect-information game where a team of $2$ collaborates to compete against the other. In its bidding phase, players bid in turn to find a good contract through a limited information channel. Based on a strong baseline agent that bids competitive bridge purely through domain-agnostic self-play, JPS improves collaboration of team players and outperforms WBridge5, a championship-winning software, by $+0.63$ IMPs (International Matching Points) per board over 1k games, substantially better than previous SoTA ($+0.41$ IMPs/b) under Double-Dummy evaluation.

Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solvers

A Generalized Training Approach for Multiagent Learning

Turbocharging Solution Concepts: Solving NEs, CEs and CCEs with Neural Equilibrium Solvers

Joint Policy Search for Multi-agent Collaboration with Imperfect Information

Neural Population Learning beyond Symmetric Zero-sum Games

A Risk-Averse Equilibrium for Multi-Agent Systems

Meta-game equilibrium for multi-agent reinforcement learning

Anytime PSRO for Two-Player Zero-Sum Games

Combining Tree-Search, Generative Models, and Nash Bargaining Concepts in Game-Theoretic Reinforcement Learning

Inducing Stackelberg Equilibrium through Spatio-Temporal Sequential Decision-Making in Multi-Agent Reinforcement Learning

Towards convergence to Nash equilibria in two-team zero-sum games

Neural Auto-Curricula in Two-Player Zero-Sum Games.

Computing Ex Ante Equilibrium in Heterogeneous Zero-Sum Team Games

Computing Ex Ante Coordinated Team-Maxmin Equilibria in Zero-Sum Multiplayer Extensive-Form Games

Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games

Fictitious Cross-Play: Learning Global Nash Equilibrium in Mixed Cooperative-Competitive Games

Team-PSRO for Learning Approximate TMECor in Large Team Games Via Cooperative Reinforcement Learning

Reinforcement Nash Equilibrium Solver

A Unified Perspective on Deep Equilibrium Finding

Oracles & Followers: Stackelberg Equilibria in Deep Multi-Agent Reinforcement Learning

Equilibrium Selection for Multi-agent Reinforcement Learning: A Unified Framework