Learning in mean field games: The fictitious play

Pierre Cardaliaguet,Saeed Hadikhanloo

DOI: https://doi.org/10.1051/cocv/2016004

2017-01-18

Abstract:Mean Field Game systems describe equilibrium configurations in differential games with infinitely many infinitesimal interacting agents. We introduce a learning procedure (similar to the Fictitious Play) for these games and show its convergence when the Mean Field Game is potential.

What problem does this paper attempt to address?

Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications

Sarah Perrin,Julien Perolat,Mathieu Laurière,Matthieu Geist,Romuald Elie,Olivier Pietquin

DOI: https://doi.org/10.48550/arXiv.2007.03458

2020-07-05

Optimization and Control

Abstract:In this paper, we deepen the analysis of continuous time Fictitious Play learning algorithm to the consideration of various finite state Mean Field Game settings (finite horizon, $\gamma$-discounted), allowing in particular for the introduction of an additional common noise. We first present a theoretical convergence analysis of the continuous time Fictitious Play process and prove that the induced exploitability decreases at a rate $O(\frac{1}{t})$. Such analysis emphasizes the use of exploitability as a relevant metric for evaluating the convergence towards a Nash equilibrium in the context of Mean Field Games. These theoretical contributions are supported by numerical experiments provided in either model-based or model-free settings. We provide hereby for the first time converging learning dynamics for Mean Field Games in the presence of common noise.
Provable Fictitious Play for General Mean-Field Games

Qiaomin Xie,Zhuoran Yang,Zhaoran Wang,Andreea Minca

DOI: https://doi.org/10.48550/arXiv.2010.04211

IF: 5.414

2020-10-08

Machine Learning

Abstract:We propose a reinforcement learning algorithm for stationary mean-field games, where the goal is to learn a pair of mean-field state and stationary policy that constitutes the Nash equilibrium. When viewing the mean-field state and the policy as two players, we propose a fictitious play algorithm which alternatively updates the mean-field state and the policy via gradient-descent and proximal policy optimization, respectively. Our algorithm is in stark contrast with previous literature which solves each single-agent reinforcement learning problem induced by the iterates mean-field states to the optimum. Furthermore, we prove that our fictitious play algorithm converges to the Nash equilibrium at a sublinear rate. To the best of our knowledge, this seems the first provably convergent single-loop reinforcement learning algorithm for mean-field games based on iterative updates of both mean-field state and policy.
Mean-Field Learning: a Survey

Hamidou Tembine,Raul Tempone,Pedro Vilanova

DOI: https://doi.org/10.48550/arXiv.1210.4657

2012-10-17

Abstract:In this paper we study iterative procedures for stationary equilibria in games with large number of players. Most of learning algorithms for games with continuous action spaces are limited to strict contraction best reply maps in which the Banach-Picard iteration converges with geometrical convergence rate. When the best reply map is not a contraction, Ishikawa-based learning is proposed. The algorithm is shown to behave well for Lipschitz continuous and pseudo-contractive maps. However, the convergence rate is still unsatisfactory. Several acceleration techniques are presented. We explain how cognitive users can improve the convergence rate based only on few number of measurements. The methodology provides nice properties in mean field games where the payoff function depends only on own-action and the mean of the mean-field (first moment mean-field games). A learning framework that exploits the structure of such games, called, mean-field learning, is proposed. The proposed mean-field learning framework is suitable not only for games but also for non-convex global optimization problems. Then, we introduce mean-field learning without feedback and examine the convergence to equilibria in beauty contest games, which have interesting applications in financial markets. Finally, we provide a fully distributed mean-field learning and its speedup versions for satisfactory solution in wireless networks. We illustrate the convergence rate improvement with numerical examples.

Machine Learning,Computer Science and Game Theory,Multiagent Systems,Dynamical Systems
Learning in anonymous nonatomic games with applications to first-order mean field games

Saeed Hadikhanloo

DOI: https://doi.org/10.48550/arXiv.1704.00378

2017-04-02

Optimization and Control

Abstract:We introduce a model of anonymous games with the player dependent action sets. We propose several learning procedures based on the well-known Fictitious Play and Online Mirror Descent and prove their convergence to equilibrium under the classical monotonicity condition. Typical examples are first-order mean field games.
Learning by Fictitious Play in Large Populations

Misha Perepelitsa

DOI: https://doi.org/10.48550/arXiv.1901.02907

2019-01-09

Computer Science and Game Theory

Abstract:We consider learning by fictitious play in a large population of agents engaged in single-play, two-person rounds of a symmetric game, and derive a mean-filed type model for the corresponding stochastic process. Using this model, we describe qualitative properties of the learning process and discuss its asymptotic behavior. Of the special interest is the comparative characteristics of the fictitious play learning with and without a memory factor. As a part of the analysis, we show that the model leads to the continuous, best-response dynamics equation of Gilboa and Matsui (1991), when all agents have similar empirical probabilities.
Learning Correlated Equilibria in Mean-Field Games

Paul Muller,Romuald Elie,Mark Rowland,Mathieu Lauriere,Julien Perolat,Sarah Perrin,Matthieu Geist,Georgios Piliouras,Olivier Pietquin,Karl Tuyls

DOI: https://doi.org/10.48550/arXiv.2208.10138

2022-08-22

Abstract:The designs of many large-scale systems today, from traffic routing environments to smart grids, rely on game-theoretic equilibrium concepts. However, as the size of an $N$-player game typically grows exponentially with $N$, standard game theoretic analysis becomes effectively infeasible beyond a low number of players. Recent approaches have gone around this limitation by instead considering Mean-Field games, an approximation of anonymous $N$-player games, where the number of players is infinite and the population's state distribution, instead of every individual player's state, is the object of interest. The practical computability of Mean-Field Nash equilibria, the most studied Mean-Field equilibrium to date, however, typically depends on beneficial non-generic structural properties such as monotonicity or contraction properties, which are required for known algorithms to converge. In this work, we provide an alternative route for studying Mean-Field games, by developing the concepts of Mean-Field correlated and coarse-correlated equilibria. We show that they can be efficiently learnt in \emph{all games}, without requiring any additional assumption on the structure of the game, using three classical algorithms. Furthermore, we establish correspondences between our notions and those already present in the literature, derive optimality bounds for the Mean-Field - $N$-player transition, and empirically demonstrate the convergence of these algorithms on simple games.

Computer Science and Game Theory,Machine Learning
Learning Discrete-Time Major-Minor Mean Field Games

Kai Cui,Gökçe Dayanıklı,Mathieu Laurière,Matthieu Geist,Olivier Pietquin,Heinz Koeppl

2023-12-18

Abstract:Recent techniques based on Mean Field Games (MFGs) allow the scalable analysis of multi-player games with many similar, rational agents. However, standard MFGs remain limited to homogeneous players that weakly influence each other, and cannot model major players that strongly influence other players, severely limiting the class of problems that can be handled. We propose a novel discrete time version of major-minor MFGs (M3FGs), along with a learning algorithm based on fictitious play and partitioning the probability simplex. Importantly, M3FGs generalize MFGs with common noise and can handle not only random exogeneous environment states but also major players. A key challenge is that the mean field is stochastic and not deterministic as in standard MFGs. Our theoretical investigation verifies both the M3FG model and its algorithmic solution, showing firstly the well-posedness of the M3FG model starting from a finite game of interest, and secondly convergence and approximation guarantees of the fictitious play algorithm. Then, we empirically verify the obtained theoretical results, ablating some of the theoretical assumptions made, and show successful equilibrium learning in three example problems. Overall, we establish a learning framework for a novel and broad class of tractable games.

Computer Science and Game Theory,Machine Learning,Multiagent Systems,Optimization and Control
Learning Graphon Mean Field Games and Approximate Nash Equilibria

Kai Cui,Heinz Koeppl

DOI: https://doi.org/10.48550/arXiv.2112.01280

2022-02-18

Abstract:Recent advances at the intersection of dense large graph limits and mean field games have begun to enable the scalable analysis of a broad class of dynamical sequential games with large numbers of agents. So far, results have been largely limited to graphon mean field systems with continuous-time diffusive or jump dynamics, typically without control and with little focus on computational methods. We propose a novel discrete-time formulation for graphon mean field games as the limit of non-linear dense graph Markov games with weak interaction. On the theoretical side, we give extensive and rigorous existence and approximation properties of the graphon mean field solution in sufficiently large systems. On the practical side, we provide general learning schemes for graphon mean field equilibria by either introducing agent equivalence classes or reformulating the graphon mean field system as a classical mean field system. By repeatedly finding a regularized optimal control solution and its generated mean field, we successfully obtain plausible approximate Nash equilibria in otherwise infeasible large dense graph games with many agents. Empirically, we are able to demonstrate on a number of examples that the finite-agent behavior comes increasingly close to the mean field behavior for our computed equilibria as the graph or system size grows, verifying our theory. More generally, we successfully apply policy gradient reinforcement learning in conjunction with sequential Monte Carlo methods.

Computer Science and Game Theory,Machine Learning,Multiagent Systems,Optimization and Control
Bounded Rationality Equilibrium Learning in Mean Field Games

Yannick Eich,Christian Fabian,Kai Cui,Heinz Koeppl

2024-11-12

Abstract:Mean field games (MFGs) tractably model behavior in large agent populations. The literature on learning MFG equilibria typically focuses on finding Nash equilibria (NE), which assume perfectly rational agents and are hence implausible in many realistic situations. To overcome these limitations, we incorporate bounded rationality into MFGs by leveraging the well-known concept of quantal response equilibria (QRE). Two novel types of MFG QRE enable the modeling of large agent populations where individuals only noisily estimate the true objective. We also introduce a second source of bounded rationality to MFGs by restricting agents' planning horizon. The resulting novel receding horizon (RH) MFGs are combined with QRE and existing approaches to model different aspects of bounded rationality in MFGs. We formally define MFG QRE and RH MFGs and compare them to existing equilibrium concepts such as entropy-regularized NE. Subsequently, we design generalized fixed point iteration and fictitious play algorithms to learn QRE and RH equilibria. After a theoretical analysis, we give different examples to evaluate the capabilities of our learning algorithms and outline practical differences between the equilibrium concepts.

Computer Science and Game Theory,Artificial Intelligence,Machine Learning,Multiagent Systems
Entropic Fictitious Play for Mean Field Optimization Problem

Fan Chen,Zhenjie Ren,Songbo Wang

2023-07-22

Abstract:We study two-layer neural networks in the mean field limit, where the number of neurons tends to infinity. In this regime, the optimization over the neuron parameters becomes the optimization over the probability measures, and by adding an entropic regularizer, the minimizer of the problem is identified as a fixed point. We propose a novel training algorithm named entropic fictitious play, inspired by the classical fictitious play in game theory for learning Nash equilibriums, to recover this fixed point, and the algorithm exhibits a two-loop iteration structure. Exponential convergence is proved in this paper and we also verify our theoretical results by simple numerical examples.

Optimization and Control,Probability
Bayesian Learning in Mean Field Games

Eran Shmaya,Bruno Ziliotto

2024-01-31

Abstract:We consider a mean-field game model where the cost functions depend on a fixed parameter, called \textit{state}, which is unknown to players. Players learn about the state from a a stream of private signals they receive throughout the game. We derive a mean field system satisfied by the equilibrium payoff of the game and prove existence of a solution under standard regularity assumptions. Additionally, we establish the uniqueness of the solution when the cost function satisfies the monotonicity assumption of Lasry and Lions at each state.

Optimization and Control,Analysis of PDEs
Reinforcement Learning for Mean Field Game

Mridul Agarwal,Vaneet Aggarwal,Arnob Ghosh,Nilay Tiwari

DOI: https://doi.org/10.48550/arXiv.1905.13357

2019-10-09

Abstract:Stochastic games provide a framework for interactions among multiple agents and enable a myriad of applications. In these games, agents decide on actions simultaneously, the state of every agent moves to the next state, and each agent receives a reward. However, finding an equilibrium (if exists) in this game is often difficult when the number of agents becomes large. This paper focuses on finding a mean-field equilibrium (MFE) in an action coupled stochastic game setting in an episodic framework. It is assumed that the impact of the other agents' can be assumed by the empirical distribution of the mean of the actions. All agents know the action distribution and employ lower-myopic best response dynamics to choose the optimal oblivious strategy. This paper proposes a posterior sampling based approach for reinforcement learning in the mean-field game, where each agent samples a transition probability from the previous transitions. We show that the policy and action distributions converge to the optimal oblivious strategy and the limiting distribution, respectively, which constitute an MFE.

Machine Learning,Artificial Intelligence
A fictitious-play finite-difference method for linearly solvable mean field games

Daisuke Inoue,Yuji Ito,Takahito Kashiwabara,Norikazu Saito,Hiroaki Yoshida

DOI: https://doi.org/10.1051/m2an/2023026

2023-03-17

ESAIM: Mathematical Modelling and Numerical Analysis

Abstract:An iterative finite difference scheme for mean field games (MFGs) is proposed. The target MFGs are derived from control problems for multidimensional systems with advection terms. For such MFGs, linearization using the Cole-Hopf transformation and iterative computation using fictitious play are introduced. This leads to an implementation-friendly algorithm that iteratively solves explicit schemes. The convergence properties of the proposed scheme are mathematically proved by tracking the error of the variable through iterations. Numerical calculations show that the proposed method works stably for both one- and two-dimensional control problems.

mathematics, applied
Learning equilibria in Cournot mean field games of controls

Fabio Camilli,Mathieu Laurière,Qing Tang

2024-10-29

Abstract:We consider Cournot mean field games of controls, a model originally developed for the production of an exhaustible resource by a continuum of producers. We prove uniqueness of the solution under general assumptions on the price function. Then, we prove convergence of a learning algorithm which gives existence of a solution to the mean field games system. The learning algorithm is implemented with a suitable finite difference discretization to get a numerical method to the solution. We supplement our theoretical analysis with several numerical examples and illustrate the impacts of model parameters.

Optimization and Control,Analysis of PDEs
Generalized Conditional Gradient and Learning in Potential Mean Field Games

Pierre Lavigne,Laurent Pfeiffer,Lavigne, Pierre,Pfeiffer, Laurent

DOI: https://doi.org/10.1007/s00245-023-10056-8

2023-10-25

Applied Mathematics & Optimization

Abstract:We investigate the resolution of second-order, potential, and monotone mean field games with the generalized conditional gradient algorithm, an extension of the Frank-Wolfe algorithm. We show that the method is equivalent to the fictitious play method. We establish rates of convergence for the optimality gap, the exploitability, and the distances of the variables to the unique solution of the mean field game, for various choices of stepsizes. In particular, we show that linear convergence can be achieved when the stepsizes are computed by linesearch.

mathematics, applied
Reinforcement Learning for Finite Space Mean-Field Type Games

Kai Shao,Jiacheng Shen,Chijie An,Mathieu Laurière

2024-09-26

Abstract:Mean field type games (MFTGs) describe Nash equilibria between large coalitions: each coalition consists of a continuum of cooperative agents who maximize the average reward of their coalition while interacting non-cooperatively with a finite number of other coalitions. Although the theory has been extensively developed, we are still lacking efficient and scalable computational methods. Here, we develop reinforcement learning methods for such games in a finite space setting with general dynamics and reward functions. We start by proving that MFTG solution yields approximate Nash equilibria in finite-size coalition games. We then propose two algorithms. The first is based on quantization of the mean-field spaces and Nash Q-learning. We provide convergence and stability analysis. We then propose an deep reinforcement learning algorithm, which can scale to larger spaces. Numerical examples on 5 environments show the scalability and the efficiency of the proposed method.

Computer Science and Game Theory,Machine Learning,Optimization and Control
An Introduction to Mean Field Game Theory

Pierre Cardaliaguet,Alessio Porretta

DOI: https://doi.org/10.1007/978-3-030-59837-2_1

2020-01-01

Lecture notes in mathematics

Abstract:These notes are an introduction to Mean Field Game (MFG) theory, which models differential games involving infinitely many interacting players. We focus here on the Partial Differential Equations (PDEs) approach to MFGs. The two main parts of the text correspond to the two emblematic equations in MFG theory: the first part is dedicated to the MFG system, while the second part is devoted to the master equation.The MFG system describes Nash equilibrium configurations in the mean field approach to differential games with infinitely many players. It consists in the coupling between a backward Hamilton-Jacobi equation (for the value function of a single player) and a forward Fokker-Planck equation (for the distribution law of the individual states). We discuss the existence and the uniqueness of the solution to the MFG system in several frameworks, depending on the presence or not of a diffusion term and on the nature of the interactions between the players (local or nonlocal coupling). We also explain how these different frameworks are related to each other. As an application, we show how to use the MFG system to find approximate Nash equilibria in games with a finite number of players and we discuss the asymptotic behavior of the MFG system.The master equation is a PDE in infinite space dimension: more precisely it is a kind of transport equation in the space of measures. The interest of this equation is that it allows to handle more complex MFG problems as, for instance, MFG problems involving a randomness affecting all the players. To analyse this equation, we first discuss the notion of derivative of maps defined on the space of measures; then we present the master equation in several frameworks (classical form, case of finite state space and case with common noise); finally we explain how to use the master equation to prove the convergence of Nash equilibria of games with finitely many players as the number of players tends to infinity.As the works on MFGs are largely inspired by P.L. Lions’ courses held at the Collège de France in the years 2007–2012, we complete the text with an appendix describing the organization of these courses.

mathematics
Independent Learning in Mean-Field Games: Satisficing Paths and Convergence to Subjective Equilibria

Bora Yongacoglu,Gürdal Arslan,Serdar Yüksel

2023-11-23

Abstract:Independent learners are agents that employ single-agent algorithms in multi-agent systems, intentionally ignoring the effect of other strategic agents. This paper studies mean-field games from a decentralized learning perspective, with two primary objectives: (i) to identify structure that can guide algorithm design, and (ii) to understand the emergent behaviour in systems of independent learners. We study a new model of partially observed mean-field games with finitely many players, local action observability, and a general observation channel for partial observations of the global state. Specific observation channels considered include (a) global observability, (b) local and mean-field observability, (c) local and compressed mean-field observability, and (d) only local observability. We establish conditions under which the control problem of a given agent is equivalent to a fully observed MDP, as well as conditions under which the control problem is equivalent only to a POMDP. Building on the connection to MDPs, we prove the existence of perfect equilibrium among memoryless stationary policies under mean-field observability. Leveraging the connection to POMDPs, we prove convergence of learning iterates obtained by independent learning agents under any of the aforementioned observation channels. We interpret the limiting values as subjective value functions, which an agent believes to be relevant to its control problem. These subjective value functions are then used to propose subjective Q-equilibrium, a new solution concept for partially observed n-player mean-field games, whose existence is proved under mean-field or global observability.We provide a decentralized learning algorithm for partially observed n-player mean-field games, and we show that it drives play to subjective Q-equilibrium by adapting the recently developed theory of satisficing paths to allow for subjectivity.

Computer Science and Game Theory
Fictitious Play via Finite Differences for Mean Field Games with Optimal Stopping

Chengfeng Shen,Yifan Luo,Zhennan Zhou

2023-10-09

Abstract:This paper considers mean field games with optimal stopping time (OSMFGs) where agents make optimal exit decisions, the coupled obstacle and Fokker-Planck equations in such models pose challenges versus classic MFGs. This paper proposes a generalized fictitious play algorithm that computes OSMFG mixed equilibria by iteratively solving pure strategy systems, i.e. approximating mixed strategies through averaging pure strategies according to a certain updating rule. The generalized fictitious play allows for a broad family of learning rates and the convergence to the mixed strategy equilibrium can be rigorously justified. The algorithm also incorporates efficient finite difference schemes of the pure strategy system, and numerical experiments demonstrate the effectiveness of the proposed method in robustly and efficiently computing mixed equilibria for OSMFGs.

Numerical Analysis
Anticipatory Fictitious Play

Alex Cloud,Albert Wang,Wesley Kerr

DOI: https://doi.org/10.48550/arXiv.2212.09941

2022-12-20

Computer Science and Game Theory

Abstract:Fictitious play is an algorithm for computing Nash equilibria of matrix games. Recently, machine learning variants of fictitious play have been successfully applied to complicated real-world games. This paper presents a simple modification of fictitious play which is a strict improvement over the original: it has the same theoretical worst-case convergence rate, is equally applicable in a machine learning context, and enjoys superior empirical performance. We conduct an extensive comparison of our algorithm with fictitious play, proving an optimal convergence rate for certain classes of games, demonstrating superior performance numerically across a variety of games, and concluding with experiments that extend these algorithms to the setting of deep multiagent reinforcement learning.

Learning in mean field games: The fictitious play

Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications

Provable Fictitious Play for General Mean-Field Games

Mean-Field Learning: a Survey

Learning in anonymous nonatomic games with applications to first-order mean field games

Learning by Fictitious Play in Large Populations

Learning Correlated Equilibria in Mean-Field Games

Learning Discrete-Time Major-Minor Mean Field Games

Learning Graphon Mean Field Games and Approximate Nash Equilibria

Bounded Rationality Equilibrium Learning in Mean Field Games

Entropic Fictitious Play for Mean Field Optimization Problem

Bayesian Learning in Mean Field Games

Reinforcement Learning for Mean Field Game

A fictitious-play finite-difference method for linearly solvable mean field games

Learning equilibria in Cournot mean field games of controls

Generalized Conditional Gradient and Learning in Potential Mean Field Games

Reinforcement Learning for Finite Space Mean-Field Type Games

An Introduction to Mean Field Game Theory

Independent Learning in Mean-Field Games: Satisficing Paths and Convergence to Subjective Equilibria

Fictitious Play via Finite Differences for Mean Field Games with Optimal Stopping

Anticipatory Fictitious Play