Abstract:Extensive-form games provide a versatile framework for modeling interactions of multiple agents subjected to imperfect observations and stochastic events. In recent years, two paradigms, policy space response oracles (PSRO) and counterfactual regret minimization (CFR), showed that extensive-form games may indeed be solved efficiently. Both of them are capable of leveraging deep neural networks to tackle the scalability issues inherent to extensive-form games and we refer to them as deep equilibrium-finding algorithms. Even though PSRO and CFR share some similarities, they are often regarded as distinct and the answer to the question of which is superior to the other remains ambiguous. Instead of answering this question directly, in this work we propose a unified perspective on deep equilibrium finding that generalizes both PSRO and CFR. Our four main contributions include: i) a novel response oracle (RO) which computes Q values as well as reaching probability values and baseline values; ii) two transform modules -- a pre-transform and a post-transform -- represented by neural networks transforming the outputs of RO to a latent additive space (LAS), and then the LAS to action probabilities for execution; iii) two average oracles -- local average oracle (LAO) and global average oracle (GAO) -- where LAO operates on LAS and GAO is used for evaluation only; and iv) a novel method inspired by fictitious play that optimizes the transform modules and average oracles, and automatically selects the optimal combination of components of the two frameworks. Experiments on Leduc poker game demonstrate that our approach can outperform both frameworks.

The Focal Quantal Response Equilibrium

Quantal Response Equilibrium and Rationalizability: Inside the Black Box

Quantal Response Equilibrium with a Continuum of Types: Characterization and Nonparametric Identification

PN Equilibrium Strategy for Matrix Games with Fuzzy Payoffs

Bounded rationality for relaxing best response and mutual consistency: the quantal hierarchy model of decision making

A Cognitive Hierarchy Model Based on Quantal Response Equilibrium

Non-strategic Econometrics (for Initial Play)

Rationalizing Irrational Beliefs

Stability of Mixed-Strategy-Based Iterative Logit Quantal Response Dynamics in Game Theory

Modeling Bounded Rationality in Capacity Allocation Games with the Quantal Response Equilibrium

Tractable Equilibrium Computation in Markov Games through Risk Aversion

Observable Perfect Equilibrium

Bounded Rationality Equilibrium Learning in Mean Field Games

Full Characterization of Quantum Correlated Equilibria

The Variable Frame Theory of Focal Points: An Experimental Study

A Unified Perspective on Deep Equilibrium Finding

Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks

Risk-Averse Equilibrium for Games

Modeling Bounded Rationality in Discretionary Lane Change with the Quantal Response Equilibrium of Game Theory

From Behavioral Theories to Econometrics: Inferring Preferences of Human Agents from Data on Repeated Interactions

Predicting human behavior in unrepeated, simultaneous-move games