Abstract:In two-player zero-sum games, if both players minimize their average external regret, then the average of the strategy profiles converges to a Nash equilibrium. For n-player general-sum games, however, theoretical guarantees for regret minimization are less understood. Nonetheless, Counterfactual Regret Minimization (CFR), a popular regret minimization algorithm for extensive-form games, has generated winning three-player Texas Hold'em agents in the Annual Computer Poker Competition (ACPC). In this paper, we provide the first set of theoretical properties for regret minimization algorithms in non-zero-sum games by proving that solutions eliminate iterative strict domination. We formally define \emph{dominated actions} in extensive-form games, show that CFR avoids iteratively strictly dominated actions and strategies, and demonstrate that removing iteratively dominated actions is enough to win a mock tournament in a small poker game. In addition, for two-player non-zero-sum games, we bound the worst case performance and show that in practice, regret minimization can yield strategies very close to equilibrium. Our theoretical advancements lead us to a new modification of CFR for games with more than two players that is more efficient and may be used to generate stronger strategies than previously possible. Furthermore, we present a new three-player Texas Hold'em poker agent that was built using CFR and a novel game decomposition method. Our new agent wins the three-player events of the 2012 ACPC and defeats the winning three-player programs from previous competitions while requiring less resources to generate than the 2011 winner. Finally, we show that our CFR modification computes a strategy of equal quality to our new agent in a quarter of the time of standard CFR using half the memory.

Actor-Critic Policy Optimization in a Large-Scale Imperfect-Information Game

ACTOR-CRITIC POLICY OPTIMIZATION IN A LARGE SCALE IMPE

Self-play Reinforcement Learning with Comprehensive Critic in Computer Games

Monte Carlo Neural Fictitious Self-Play: Achieve Approximate Nash equilibrium of Imperfect-Information Games.

RM-FSP: Regret Minimization Optimizes Neural Fictitious Self-Play

Optimal Policy of Multiplayer Poker via Actor-Critic Reinforcement Learning

A Policy-Gradient Approach to Solving Imperfect-Information Games with Iterate Convergence

Optimize Neural Fictitious Self-Play in Regret Minimization Thinking

Efficient Competitive Self-Play Policy Optimization

A Monte Carlo Neural Fictitious Self-Play approach to approximate Nash Equilibrium in imperfect-information dynamic games

The Advantage Regret-Matching Actor-Critic

Regret Minimization in Non-Zero-Sum Games with Applications to Building Champion Multiplayer Computer Poker Agents

Extensive Game Decision Based on the PPO-CFR Algorithm under Incomplete Information

Monte Carlo Neural Fictitious Self-Play: Approach to Approximate Nash equilibrium of Imperfect-Information Games

Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

A Unified Perspective on Deep Equilibrium Finding

CFR-p: Counterfactual Regret Minimization with Hierarchical Policy Abstraction, and its Application to Two-player Mahjong

Scalable sub-game solving for imperfect-information games

Deep Counterfactual Regret Minimization

Actor-Critic Reinforcement Learning Algorithms for Mean Field Games in Continuous Time, State and Action Spaces

Finding nash equilibrium for imperfect information games via fictitious play based on local regret minimization