Abstract:Recently, Daskalakis, Fishelson, and Golowich (DFG) (NeurIPS`21) showed that if all agents in a multi-player general-sum normal-form game employ Optimistic Multiplicative Weights Update (OMWU), the external regret of every player is $O(\textrm{polylog}(T))$ after $T$ repetitions of the game. We extend their result from external regret to internal regret and swap regret, thereby establishing uncoupled learning dynamics that converge to an approximate correlated equilibrium at the rate of $\tilde{O}(T^{-1})$. This substantially improves over the prior best rate of convergence for correlated equilibria of $O(T^{-3/4})$ due to Chen and Peng (NeurIPS`20), and it is optimal -- within the no-regret framework -- up to polylogarithmic factors in $T$. To obtain these results, we develop new techniques for establishing higher-order smoothness for learning dynamics involving fixed point operations. Specifically, we establish that the no-internal-regret learning dynamics of Stoltz and Lugosi (Mach Learn`05) are equivalently simulated by no-external-regret dynamics on a combinatorial space. This allows us to trade the computation of the stationary distribution on a polynomial-sized Markov chain for a (much more well-behaved) linear transformation on an exponential-sized set, enabling us to leverage similar techniques as DFG to near-optimally bound the internal regret. Moreover, we establish an $O(\textrm{polylog}(T))$ no-swap-regret bound for the classic algorithm of Blum and Mansour (BM) (JMLR`07). We do so by introducing a technique based on the Cauchy Integral Formula that circumvents the more limited combinatorial arguments of DFG. In addition to shedding clarity on the near-optimal regret guarantees of BM, our arguments provide insights into the various ways in which the techniques by DFG can be extended and leveraged in the analysis of more involved learning algorithms.

Near-Optimal Policy Optimization for Correlated Equilibrium in General-Sum Markov Games

Near-Optimal Last-iterate Convergence of Policy Optimization in Zero-sum Polymatrix Markov Games

Optimal Correlated Equilibria in General-Sum Extensive-Form Games: Fixed-Parameter Algorithms, Hardness, and Two-Sided Column-Generation

Near-Optimal No-Regret Learning for Correlated Equilibria in Multi-Player General-Sum Games

Robust optimal policies for team Markov games

Empirical Policy Optimization for n-Player Markov Games

Efficient Learning and Computation of Linear Correlated Equilibrium in General Convex Games

Efficiently Solving Turn-Taking Stochastic Games with Extensive-Form Correlation

Empirical Policy Optimization for <i>n</i>-Player Markov Games

A Coupled Optimization Framework for Correlated Equilibria in Normal-Form Game

Convergence of Policy Gradient Methods for Nash Equilibria in General-sum Stochastic Games

Leveraging Team Correlation for Approximating Equilibrium in Two-Team Zero-Sum Games

Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence Beyond the Minty Property

Provably Efficient Fictitious Play Policy Optimization for Zero-Sum Markov Games with Structured Transitions

Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games

Policy Optimization finds Nash Equilibrium in Regularized General-Sum LQ Games

Fast swap regret minimization and applications to approximate correlated equilibria

Faster Algorithms for Optimal Ex-Ante Coordinated Collusive Strategies in Extensive-Form Zero-Sum Games

Doubly Optimal No-Regret Learning in Monotone Games

A Policy-Gradient Approach to Solving Imperfect-Information Games with Iterate Convergence

When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?