Abstract:Imperfect Information Games (IIGs) offer robust models for scenarios where decision-makers face uncertainty or lack complete information. Counterfactual Regret Minimization (CFR) has been one of the most successful family of algorithms for tackling IIGs. The integration of skill-based strategy learning with CFR could potentially mirror more human-like decision-making process and enhance the learning performance for complex IIGs. It enables the learning of a hierarchical strategy, wherein low-level components represent skills for solving subgames and the high-level component manages the transition between skills. In this paper, we introduce the first hierarchical version of Deep CFR (HDCFR), an innovative method that boosts learning efficiency in tasks involving extensively large state spaces and deep game trees. A notable advantage of HDCFR over previous works is its ability to facilitate learning with predefined (human) expertise and foster the acquisition of skills that can be transferred to similar tasks. To achieve this, we initially construct our algorithm on a tabular setting, encompassing hierarchical CFR updating rules and a variance-reduced Monte Carlo sampling extension. Notably, we offer the theoretical justifications, including the convergence rate of the proposed updating rule, the unbiasedness of the Monte Carlo regret estimator, and ideal criteria for effective variance reduction. Then, we employ neural networks as function approximators and develop deep learning objectives to adapt our proposed algorithms for large-scale tasks, while maintaining the theoretical support.

Parallel Counterfactual Regret Minimization in Crowdsourcing Imperfect-information Expanded Game

Hierarchical Deep Counterfactual Regret Minimization

Solving Imperfect-Information Games Via Exponential Counterfactual Regret Minimization

Double Neural Counterfactual Regret Minimization.

GPU-Accelerated Counterfactual Regret Minimization

Lazy-CFR: a Fast Regret Minimization Algorithm for Extensive Games with Imperfect Information.

Efficient CFR for Imperfect Information Games with Instant Updates

Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

Model-Free Neural Counterfactual Regret Minimization with Bootstrap Learning

Deep Counterfactual Regret Minimization

Imization for extensive games with imperfect information

Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Imperfect Information

Minimizing Weighted Counterfactual Regret with Optimistic Online Mirror Descent

Scalable sub-game solving for imperfect-information games

D2CFR: Minimize Counterfactual Regret With Deep Dueling Neural Network

A Fast-Convergence Method of Monte Carlo Counterfactual Regret Minimization for Imperfect Information Dynamic Games

RLCFR: Minimize counterfactual regret by deep reinforcement learning

Equivalence Analysis between Counterfactual Regret Minimization and Online Mirror Descent

Accelerating Nash Equilibrium Convergence in Monte Carlo Settings Through Counterfactual Value Based Fictitious Play

Extensive Game Decision Based on the PPO-CFR Algorithm under Incomplete Information

CFR-p: Counterfactual Regret Minimization with Hierarchical Policy Abstraction, and its Application to Two-player Mahjong