Abstract:A major challenge in multi-agent systems is that the system complexity grows dramatically with the number of agents as well as the size of their action spaces, which is typical in real world scenarios such as autonomous vehicles, robotic teams, network routing, etc. It is hence in imminent need to design decentralized or independent algorithms where the update of each agent is only based on their local observations without the need of introducing complex communication/coordination mechanisms. In this work, we study the finite-time convergence of independent entropy-regularized natural policy gradient (NPG) methods for potential games, where the difference in an agent's utility function due to unilateral deviation matches exactly that of a common potential function. The proposed entropy-regularized NPG method enables each agent to deploy symmetric, decentralized, and multiplicative updates according to its own payoff. We show that the proposed method converges to the quantal response equilibrium (QRE)-the equilibrium to the entropy-regularized game-at a sublinear rate, which is independent of the size of the action space and grows at most sublinearly with the number of agents. Appealingly, the convergence rate further becomes independent with the number of agents for the important special case of identical-interest games, leading to the first method that converges at a dimension-free rate. Our approach can be used as a smoothing technique to find an approximate Nash equilibrium (NE) of the unregularized problem without assuming that stationary policies are isolated.

Global Convergence of Natural Policy Gradient with Hessian-Aided Momentum Variance Reduction

Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction

Decentralized Natural Policy Gradient with Variance Reduction for Collaborative Multi-Agent Reinforcement Learning

Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization.

Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policies

Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs

Accelerated Policy Gradient: On the Convergence Rates of the Nesterov Momentum for Reinforcement Learning

Fast Stochastic Policy Gradient: Negative Momentum for Reinforcement Learning

Hessian Aided Policy Gradient

Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updates

Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Games

Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes

Decentralized Multi-Task Reinforcement Learning Policy Gradient Method with Momentum over Networks.

Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

A Globally Convergent Gradient Method with Momentum

A Distributed Adaptive Policy Gradient Method Based on Momentum for Multi-Agent Reinforcement Learning

Global Convergence of Receding-Horizon Policy Search in Learning Estimator Designs

Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processes

Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games

Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control

Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization