Abstract:Non-cooperative and cooperative games with a very large number of players have many applications but remain generally intractable when the number of players increases. Introduced by Lasry and Lions, and Huang, Caines and Malhamé, Mean Field Games (MFGs) rely on a mean-field approximation to allow the number of players to grow to infinity. Traditional methods for solving these games generally rely on solving partial or stochastic differential equations with a full knowledge of the model. Recently, Reinforcement Learning (RL) has appeared promising to solve complex problems at scale. The combination of RL and MFGs is promising to solve games at a very large scale both in terms of population size and environment complexity. In this survey, we review the quickly growing recent literature on RL methods to learn equilibria and social optima in MFGs. We first identify the most common settings (static, stationary, and evolutive) of MFGs. We then present a general framework for classical iterative methods (based on best-response computation or policy evaluation) to solve MFGs in an exact way. Building on these algorithms and the connection with Markov Decision Processes, we explain how RL can be used to learn MFG solutions in a model-free way. Last, we present numerical illustrations on a benchmark problem, and conclude with some perspectives.

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is the difficult problem of solving in large - scale non - cooperative and cooperative games among players. Specifically, when the number of players increases, traditional game - solving methods become intractable. To address this issue, the paper introduces the concept of "Mean Field Games (MFGs)". **Key background information**: 1. **Complexity of large - scale games**: - In non - cooperative and cooperative games with a large number of players participating, traditional solving methods are difficult to handle. - For example, when the number of players increases, calculating the optimal strategy for each player becomes extremely complex. 2. **Introduction of Mean Field Games (MFGs)**: - Mean Field Games were introduced by Lasry and Lions (2007) and Huang et al. (2006). - MFGs simplify model analysis by using the mean - field approximation to extend the number of players to infinity. - This method assumes that the player group is homogeneous and that each player only interacts with the overall player distribution, rather than directly interacting with other individual players. 3. **Application of Reinforcement Learning (RL)**: - Traditional MFG - solving methods rely on solving partial differential equations or stochastic differential equations and require complete knowledge of the model. - Recent research shows that RL can solve complex problems without complete model knowledge, especially in large - scale environments. - Combining RL with MFG can effectively solve the problems of large - scale multi - agent systems, both in terms of population size and environmental complexity. **Main objectives of the paper**: - **Review existing literature**: Summarize the rapidly developing literature in recent years on using RL methods to learn Nash equilibria and socially optimal solutions in MFGs. - **Introduce different settings**: Explain the several most common settings in MFGs (static, stationary, evolutionary) and describe their differences and connections. - **Provide a solution framework**: Demonstrate how to accurately solve MFGs using classical iterative methods and explain the relationship between these algorithms and Markov Decision Processes (MDP). - **Explore the application of RL**: Based on the above algorithms and the connection between MDP and RL, explain how to use model - free RL methods to learn the solutions of MFGs. - **Numerical experiments and prospects**: Verify the effectiveness of the method through numerical experiments and look ahead to future research directions. In summary, this paper aims to explore how to effectively solve the difficult problem of solving large - scale multi - agent systems within the framework of Mean Field Games by using reinforcement learning techniques.

Learning in Mean Field Games: A Survey

Bounded Rationality Equilibrium Learning in Mean Field Games

Mean-Field Learning: a Survey

A General Framework for Learning Mean-Field Games

Reinforcement Learning for Mean Field Games with Strategic Complementarities

Learning Deep Mean Field Games for Modeling Large Population Behavior

Learning Correlated Equilibria in Mean-Field Games

From Nash Equilibrium to Social Optimum and vice versa: a Mean Field Perspective

Model-Free Reinforcement Learning for Mean Field Games

Model-free Reinforcement Learning for Non-stationary Mean Field Games

Reinforcement Learning for Finite Space Mean-Field Type Games

Learning Discrete-Time Major-Minor Mean Field Games

Reinforcement Learning for Mean Field Game

A Single Online Agent Can Efficiently Learn Mean Field Games

Reinforcement Learning for Non-stationary Discrete-Time Linear–Quadratic Mean-Field Games in Multiple Populations

MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games

A Machine Learning Method for Stackelberg Mean Field Games

When is Mean-Field Reinforcement Learning Tractable and Relevant?

Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning

Learning Graphon Mean Field Games and Approximate Nash Equilibria

Learning Mean Field Games on Sparse Graphs: A Hybrid Graphex Approach