Abstract:Reinforcement learning (RL) has recently achieved tremendous successes in many artificial intelligence applications. Many of the forefront applications of RL involve multiple agents, e.g., playing chess and Go games, autonomous driving, and robotics. Unfortunately, the framework upon which classical RL builds is inappropriate for multi-agent learning, as it assumes an agent's environment is stationary and does not take into account the adaptivity of other agents. In this review paper, we present the model of stochastic games for multi-agent learning in dynamic environments. We focus on the development of simple and independent learning dynamics for stochastic games: each agent is myopic and chooses best-response type actions to other agents' strategy without any coordination with her opponent. There has been limited progress on developing convergent best-response type independent learning dynamics for stochastic games. We present our recently proposed simple and independent learning dynamics that guarantee convergence in zero-sum stochastic games, together with a review of other contemporaneous algorithms for dynamic multi-agent learning in this setting. Along the way, we also reexamine some classical results from both the game theory and RL literature, to situate both the conceptual contributions of our independent learning dynamics, and the mathematical novelties of our analysis. We hope this review paper serves as an impetus for the resurgence of studying independent and natural learning dynamics in game theory, for the more challenging settings with a dynamic environment.

On Passivity and Reinforcement Learning in Finite Games.

On Passivity, Reinforcement Learning and Higher-Order Learning in Multi-Agent Finite Games

Penalty-Regulated Dynamics and Robust Learning Procedures in Games

Passivity-based Gradient-Play Dynamics for Distributed Generalized Nash Equilibrium Seeking

Counterclockwise Dissipativity, Potential Games and Evolutionary Nash Equilibrium Learning

On convergence rates of game theoretic reinforcement learning algorithms

Learning to Play General-Sum Games against Multiple Boundedly Rational Agents

Reinforcement Learning for Finite Space Mean-Field Type Games

Passivity Tools for Hybrid Learning Rules in Large Populations

Independent and Decentralized Learning in Markov Potential Games

Game-theoretical control with continuous action sets

Independent Learning in Stochastic Games

Uncoupled and Convergent Learning in Monotone Games under Bandit Feedback

A unified stochastic approximation framework for learning in games

On the Rate of Convergence of Continuous-Time Game Dynamics in N-Player Potential Games

On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning.

On Gradient-Based Learning in Continuous Games

Steering control of payoff-maximizing players in adaptive learning dynamics

Evolutionary Games on Infinite Strategy Sets: Convergence to Nash Equilibria via Dissipativity

A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning

A Probabilistic Approach to Discounted Infinite Horizon and Invariant Mean Field Games