Abstract:In two-player zero-sum games, if both players minimize their average external regret, then the average of the strategy profiles converges to a Nash equilibrium. For n-player general-sum games, however, theoretical guarantees for regret minimization are less understood. Nonetheless, Counterfactual Regret Minimization (CFR), a popular regret minimization algorithm for extensive-form games, has generated winning three-player Texas Hold'em agents in the Annual Computer Poker Competition (ACPC). In this paper, we provide the first set of theoretical properties for regret minimization algorithms in non-zero-sum games by proving that solutions eliminate iterative strict domination. We formally define \emph{dominated actions} in extensive-form games, show that CFR avoids iteratively strictly dominated actions and strategies, and demonstrate that removing iteratively dominated actions is enough to win a mock tournament in a small poker game. In addition, for two-player non-zero-sum games, we bound the worst case performance and show that in practice, regret minimization can yield strategies very close to equilibrium. Our theoretical advancements lead us to a new modification of CFR for games with more than two players that is more efficient and may be used to generate stronger strategies than previously possible. Furthermore, we present a new three-player Texas Hold'em poker agent that was built using CFR and a novel game decomposition method. Our new agent wins the three-player events of the 2012 ACPC and defeats the winning three-player programs from previous competitions while requiring less resources to generate than the 2011 winner. Finally, we show that our CFR modification computes a strategy of equal quality to our new agent in a quarter of the time of standard CFR using half the memory.

Efficient CFR for Imperfect Information Games with Instant Updates

Imization for extensive games with imperfect information

Lazy-CFR: a Fast Regret Minimization Algorithm for Extensive Games with Imperfect Information.

Scalable sub-game solving for imperfect-information games

A Fast-Convergence Method of Monte Carlo Counterfactual Regret Minimization for Imperfect Information Dynamic Games

Deep Counterfactual Regret Minimization

Faster Game Solving via Hyperparameter Schedules

GPU-Accelerated Counterfactual Regret Minimization

Accelerating Nash Equilibrium Convergence in Monte Carlo Settings Through Counterfactual Value Based Fictitious Play

Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

CFR-p: Counterfactual Regret Minimization with Hierarchical Policy Abstraction, and its Application to Two-player Mahjong

Regret Minimization in Non-Zero-Sum Games with Applications to Building Champion Multiplayer Computer Poker Agents

RLCFR: Minimize counterfactual regret by deep reinforcement learning

D2CFR: Minimize Counterfactual Regret With Deep Dueling Neural Network

Double Neural Counterfactual Regret Minimization.

Modeling Other Players with Bayesian Beliefs for Games with Incomplete Information

Parallel Counterfactual Regret Minimization in Crowdsourcing Imperfect-information Expanded Game

RM-FSP: Regret Minimization Optimizes Neural Fictitious Self-Play

Kdb-D2CFR: Solving Multiplayer imperfect-information games with knowledge distillation-based DeepCFR

No-Regret Learning in Extensive-Form Games with Imperfect Recall

Online Convex Optimization for Sequential Decision Processes and Extensive-Form Games