Abstract:Extensive-form games provide a versatile framework for modeling interactions of multiple agents subjected to imperfect observations and stochastic events. In recent years, two paradigms, policy space response oracles (PSRO) and counterfactual regret minimization (CFR), showed that extensive-form games may indeed be solved efficiently. Both of them are capable of leveraging deep neural networks to tackle the scalability issues inherent to extensive-form games and we refer to them as deep equilibrium-finding algorithms. Even though PSRO and CFR share some similarities, they are often regarded as distinct and the answer to the question of which is superior to the other remains ambiguous. Instead of answering this question directly, in this work we propose a unified perspective on deep equilibrium finding that generalizes both PSRO and CFR. Our four main contributions include: i) a novel response oracle (RO) which computes Q values as well as reaching probability values and baseline values; ii) two transform modules -- a pre-transform and a post-transform -- represented by neural networks transforming the outputs of RO to a latent additive space (LAS), and then the LAS to action probabilities for execution; iii) two average oracles -- local average oracle (LAO) and global average oracle (GAO) -- where LAO operates on LAS and GAO is used for evaluation only; and iv) a novel method inspired by fictitious play that optimizes the transform modules and average oracles, and automatically selects the optimal combination of components of the two frameworks. Experiments on Leduc poker game demonstrate that our approach can outperform both frameworks.

Ensemble Strategy Learning for Imperfect Information Games

Student of Games: A unified learning algorithm for both perfect and imperfect information games

Score-Based Equilibrium Learning in Multi-Player Finite Games with Imperfect Information

Scalable sub-game solving for imperfect-information games

Monte Carlo Neural Fictitious Self-Play: Achieve Approximate Nash equilibrium of Imperfect-Information Games.

Near-Optimal Learning of Extensive-Form Games with Imperfect Information

Playing Extensive Games with Learning of Opponent's Cognition

In-Context Exploiter for Extensive-Form Games

A Linear Programming-Based Reinforcement Learning Mechanism for Incomplete-Information Games

Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

Learning Strategy Representation for Imitation Learning in Multi-Agent Games

Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

Neural Auto-Curricula

A Unified Perspective on Deep Equilibrium Finding

Neural Auto-Curricula in Two-Player Zero-Sum Games.

A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning

Opponent Modeling in Multiplayer Imperfect-Information Games

Posterior Sampling for Multi-Agent Reinforcement Learning: Solving Extensive Games with Imperfect Information

Evolutionary Reinforcement Learning with Action Sequence Search for Imperfect Information Games

Social Optimum Equilibrium Selection for Distributed Multi-Agent Optimization

A Policy-Gradient Approach to Solving Imperfect-Information Games with Iterate Convergence