Abstract:In two-player zero-sum games, if both players minimize their average external regret, then the average of the strategy profiles converges to a Nash equilibrium. For n-player general-sum games, however, theoretical guarantees for regret minimization are less understood. Nonetheless, Counterfactual Regret Minimization (CFR), a popular regret minimization algorithm for extensive-form games, has generated winning three-player Texas Hold'em agents in the Annual Computer Poker Competition (ACPC). In this paper, we provide the first set of theoretical properties for regret minimization algorithms in non-zero-sum games by proving that solutions eliminate iterative strict domination. We formally define \emph{dominated actions} in extensive-form games, show that CFR avoids iteratively strictly dominated actions and strategies, and demonstrate that removing iteratively dominated actions is enough to win a mock tournament in a small poker game. In addition, for two-player non-zero-sum games, we bound the worst case performance and show that in practice, regret minimization can yield strategies very close to equilibrium. Our theoretical advancements lead us to a new modification of CFR for games with more than two players that is more efficient and may be used to generate stronger strategies than previously possible. Furthermore, we present a new three-player Texas Hold'em poker agent that was built using CFR and a novel game decomposition method. Our new agent wins the three-player events of the 2012 ACPC and defeats the winning three-player programs from previous competitions while requiring less resources to generate than the 2011 winner. Finally, we show that our CFR modification computes a strategy of equal quality to our new agent in a quarter of the time of standard CFR using half the memory.

Kdb-D2CFR: Solving Multiplayer imperfect-information games with knowledge distillation-based DeepCFR

D2CFR: Minimize Counterfactual Regret With Deep Dueling Neural Network

Scalable sub-game solving for imperfect-information games

Deep Counterfactual Regret Minimization

Monte Carlo Neural Fictitious Self-Play: Achieve Approximate Nash equilibrium of Imperfect-Information Games.

Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

Modeling Other Players with Bayesian Beliefs for Games with Incomplete Information

Double Neural Counterfactual Regret Minimization.

Efficient CFR for Imperfect Information Games with Instant Updates

CFR-p: Counterfactual Regret Minimization with Hierarchical Policy Abstraction, and its Application to Two-player Mahjong

An Efficient Deep Reinforcement Learning Algorithm for Solving Imperfect Information Extensive-Form Games.

A Unified Perspective on Deep Equilibrium Finding

RM-FSP: Regret Minimization Optimizes Neural Fictitious Self-Play

A Survey of Nash Equilibrium Strategy Solving Based on CFR

Regret Minimization in Non-Zero-Sum Games with Applications to Building Champion Multiplayer Computer Poker Agents

RL-CFR: Improving Action Abstraction for Imperfect Information Extensive-Form Games with Reinforcement Learning

Imperfect Information Game in Multiplayer No-limit Texas Hold’em Based on Mean Approximation and Deep CFVnet

RLCFR: Minimize counterfactual regret by deep reinforcement learning

Lazy-CFR: a Fast Regret Minimization Algorithm for Extensive Games with Imperfect Information.

Monte Carlo Neural Fictitious Self-Play: Approach to Approximate Nash equilibrium of Imperfect-Information Games

Actor-Critic Policy Optimization in a Large-Scale Imperfect-Information Game