Abstract:No-regret algorithms are popular for learning Nash equilibrium (NE) in two-player zero-sum normal-form games (NFGs) and extensive-form games (EFGs). Many recent works consider the last-iterate convergence no-regret algorithms. Among them, the two most famous algorithms are Optimistic Gradient Descent Ascent (OGDA) and Optimistic Multiplicative Weight Update (OMWU). However, OGDA has high per-iteration complexity. OMWU exhibits a lower per-iteration complexity but poorer empirical performance, and its convergence holds only when NE is unique. Recent works propose a Reward Transformation (RT) framework for MWU, which removes the uniqueness condition and achieves competitive performance with OMWU. Unfortunately, RT-based algorithms perform worse than OGDA under the same number of iterations, and their convergence guarantee is based on the continuous-time feedback assumption, which does not hold in most scenarios. To address these issues, we provide a closer analysis of the RT framework, which holds for both continuous and discrete-time feedback. We demonstrate that the essence of the RT framework is to transform the problem of learning NE in the original game into a series of strongly convex-concave optimization problems (SCCPs). We show that the bottleneck of RT-based algorithms is the speed of solving SCCPs. To improve the their empirical performance, we design a novel transformation method to enable the SCCPs can be solved by Regret Matching+ (RM+), a no-regret algorithm with better empirical performance, resulting in Reward Transformation RM+ (RTRM+). RTRM+ enjoys last-iterate convergence under the discrete-time feedback setting. Using the counterfactual regret decomposition framework, we propose Reward Transformation CFR+ (RTCFR+) to extend RTRM+ to EFGs. Experimental results show that our algorithms significantly outperform existing last-iterate convergence algorithms and RM+ (CFR+).

Equilibrium Finding in Normal-Form Games Via Greedy Regret Minimization

Integrating Dynamic Weighted Approach with Fictitious Play and Pure Counterfactual Regret Minimization for Equilibrium Finding

Learning not to Regret

Iterated Regret Minimization: A More Realistic Solution Concept

Regret Pruning for Learning Equilibria in Simulation-Based Games

Online Convex Optimization for Sequential Decision Processes and Extensive-Form Games

RM-FSP: Regret Minimization Optimizes Neural Fictitious Self-Play

Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descent

Is Learning in Games Good for the Learners?

Regret Minimization in Non-Zero-Sum Games with Applications to Building Champion Multiplayer Computer Poker Agents

Generalized Bandit Regret Minimizer Framework in Imperfect Information Extensive-Form Game.

Evolutionary Dynamics and $Φ$-Regret Minimization in Games

Doubly Optimal No-Regret Learning in Monotone Games

Efficient Last-iterate Convergence Algorithms in Solving Games

Sparsified Linear Programming for Zero-Sum Equilibrium Finding

Near-Optimal Learning of Extensive-Form Games with Imperfect Information

Taming Equilibrium Bias in Risk-Sensitive Multi-Agent Reinforcement Learning

Regret-Minimizing Double Oracle for Extensive-Form Games

Geometrical Regret Matching

Equilibria in Repeated Games under No-Regret with Dynamic Benchmarks

Last-Iterate Convergence Properties of Regret-Matching Algorithms in Games