Abstract:A major challenge in multi-agent systems is that the system complexity grows dramatically with the number of agents as well as the size of their action spaces, which is typical in real world scenarios such as autonomous vehicles, robotic teams, network routing, etc. It is hence in imminent need to design decentralized or independent algorithms where the update of each agent is only based on their local observations without the need of introducing complex communication/coordination mechanisms. In this work, we study the finite-time convergence of independent entropy-regularized natural policy gradient (NPG) methods for potential games, where the difference in an agent's utility function due to unilateral deviation matches exactly that of a common potential function. The proposed entropy-regularized NPG method enables each agent to deploy symmetric, decentralized, and multiplicative updates according to its own payoff. We show that the proposed method converges to the quantal response equilibrium (QRE)-the equilibrium to the entropy-regularized game-at a sublinear rate, which is independent of the size of the action space and grows at most sublinearly with the number of agents. Appealingly, the convergence rate further becomes independent with the number of agents for the important special case of identical-interest games, leading to the first method that converges at a dimension-free rate. Our approach can be used as a smoothing technique to find an approximate Nash equilibrium (NE) of the unregularized problem without assuming that stationary policies are isolated.

Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control

Convergence Rate of Primal-Dual Approach to Constrained Reinforcement Learning with Softmax Policy

On the Convergence of Discounted Policy Gradient Methods

Convergence of Policy Gradient for Stochastic Linear-Quadratic Control Problem in Infinite Horizon

A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee

Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators

Elementary Analysis of Policy Gradient Methods

A Policy-Gradient Approach to Solving Imperfect-Information Games with Iterate Convergence

On the Linear Convergence of Natural Policy Gradient Algorithm

Convergence of Policy Gradient Methods for Finite-Horizon Exploratory Linear-Quadratic Control Problems

Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs

On the Global Convergence of Risk-Averse Policy Gradient Methods with Expected Conditional Risk Measures

On the Second-Order Convergence of Biased Policy Gradient Algorithms

Policy Gradient Method For Robust Reinforcement Learning

Stochastic Cubic-Regularized Policy Gradient Method

Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization.

Convergence of Policy Gradient Methods for Nash Equilibria in General-sum Stochastic Games

Gradient play in stochastic games: stationary points, convergence, and sample complexity

Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries

Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action

Globally Convergent Policy Gradient Methods for Linear Quadratic Control of Partially Observed Systems