Abstract:In two-player zero-sum games, if both players minimize their average external regret, then the average of the strategy profiles converges to a Nash equilibrium. For n-player general-sum games, however, theoretical guarantees for regret minimization are less understood. Nonetheless, Counterfactual Regret Minimization (CFR), a popular regret minimization algorithm for extensive-form games, has generated winning three-player Texas Hold'em agents in the Annual Computer Poker Competition (ACPC). In this paper, we provide the first set of theoretical properties for regret minimization algorithms in non-zero-sum games by proving that solutions eliminate iterative strict domination. We formally define \emph{dominated actions} in extensive-form games, show that CFR avoids iteratively strictly dominated actions and strategies, and demonstrate that removing iteratively dominated actions is enough to win a mock tournament in a small poker game. In addition, for two-player non-zero-sum games, we bound the worst case performance and show that in practice, regret minimization can yield strategies very close to equilibrium. Our theoretical advancements lead us to a new modification of CFR for games with more than two players that is more efficient and may be used to generate stronger strategies than previously possible. Furthermore, we present a new three-player Texas Hold'em poker agent that was built using CFR and a novel game decomposition method. Our new agent wins the three-player events of the 2012 ACPC and defeats the winning three-player programs from previous competitions while requiring less resources to generate than the 2011 winner. Finally, we show that our CFR modification computes a strategy of equal quality to our new agent in a quarter of the time of standard CFR using half the memory.

HORSE-CFR: Hierarchical Opponent Reasoning for Safe Exploitation Counterfactual Regret Minimization

RM-FSP: Regret Minimization Optimizes Neural Fictitious Self-Play

Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

Modeling Other Players with Bayesian Beliefs for Games with Incomplete Information

Safe Opponent-Exploitation Subgame Refinement

Extensive Game Decision Based on the PPO-CFR Algorithm under Incomplete Information

Hierarchical Deep Counterfactual Regret Minimization

Monte Carlo Neural Fictitious Self-Play: Achieve Approximate Nash equilibrium of Imperfect-Information Games.

RL-CFR: Improving Action Abstraction for Imperfect Information Extensive-Form Games with Reinforcement Learning

Regret Minimization in Non-Zero-Sum Games with Applications to Building Champion Multiplayer Computer Poker Agents

Modeling Rationality: Toward Better Performance Against Unknown Agents in Sequential Games

Double Neural Counterfactual Regret Minimization.

CFR-p: Counterfactual Regret Minimization with Hierarchical Policy Abstraction, and its Application to Two-player Mahjong

D2CFR: Minimize Counterfactual Regret With Deep Dueling Neural Network

An Efficient Deep Reinforcement Learning Algorithm for Solving Imperfect Information Extensive-Form Games.

Exploiting a No-Regret Opponent in Repeated Zero-Sum Games

Scalable sub-game solving for imperfect-information games

Optimize Neural Fictitious Self-Play in Regret Minimization Thinking

Solving Imperfect-Information Games Via Exponential Counterfactual Regret Minimization

Improving Agent Decision Payoffs via a New Framework of Opponent Modeling

The Advantage Regret-Matching Actor-Critic