Abstract:Regret minimization has proved to be a versatile tool for tree-form sequential decision making and extensive-form games. In large two-player zero-sum imperfect-information games, modern extensions of counterfactual regret minimization (CFR) are currently the practical state of the art for computing a Nash equilibrium. Most regret-minimization algorithms for tree-form sequential decision making, including CFR, require (i) an exact model of the player's decision nodes, observation nodes, and how they are linked, and (ii) full knowledge, at all times t, about the payoffs -- even in parts of the decision space that are not encountered at time t. Recently, there has been growing interest towards relaxing some of those restrictions and making regret minimization applicable to settings for which reinforcement learning methods have traditionally been used -- for example, those in which only black-box access to the environment is available. We give the first, to our knowledge, regret-minimization algorithm that guarantees sublinear regret with high probability even when requirement (i) -- and thus also (ii) -- is dropped. We formalize an online learning setting in which the strategy space is not known to the agent and gets revealed incrementally whenever the agent encounters new decision points. We give an efficient algorithm that achieves $O(T^{3/4})$ regret with high probability for that setting, even when the agent faces an adversarial environment. Our experiments show it significantly outperforms the prior algorithms for the problem, which do not have such guarantees. It can be used in any application for which regret minimization is useful: approximating Nash equilibrium or quantal response equilibrium, approximating coarse correlated equilibrium in multi-player games, learning a best response, learning safe opponent exploitation, and online play against an unknown opponent/environment.

A New Epistemic Logic Model of Regret Games.

A Unified Epistemic Analysis Of Iterated Elimination Algorithms From Regret Viewpoint

Ignorance is Bliss: A Game of Regret

Iterated Regret Minimization: A More Realistic Solution Concept

Erratum: "rational dynamics and epistemic logic in games"

Rational Dynamics And Epistemic Logic In Games (Vol 9, Pg 13, 2007)

Rationalizing Irrational Beliefs

Ambiguous Games Played by Players with Ambiguity Aversion and Minimax Regret.

Rational Dynamics and Epistemic Logic in Games

Solving Imperfect-Information Games Via Exponential Counterfactual Regret Minimization

From Rules to Runs:A Dynamic Epistemic Take on Imperfect Information Games

RM-FSP: Regret Minimization Optimizes Neural Fictitious Self-Play

Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games

Optimize Neural Fictitious Self-Play in Regret Minimization Thinking

Games with ambiguous payoffs and played by ambiguity and regret minimising players

A Game Model with Private Goal and Belief

Existence of open-loop equilibria in differential games with impulsive effects suffered by logic choice

Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

Reactive Synthesis Without Regret

Finding nash equilibrium for imperfect information games via fictitious play based on local regret minimization

Regret Pruning for Learning Equilibria in Simulation-Based Games