Learning Graphon Mean Field Games and Approximate Nash Equilibria

Kai Cui,Heinz Koeppl
DOI: https://doi.org/10.48550/arXiv.2112.01280
2022-02-18
Abstract:Recent advances at the intersection of dense large graph limits and mean field games have begun to enable the scalable analysis of a broad class of dynamical sequential games with large numbers of agents. So far, results have been largely limited to graphon mean field systems with continuous-time diffusive or jump dynamics, typically without control and with little focus on computational methods. We propose a novel discrete-time formulation for graphon mean field games as the limit of non-linear dense graph Markov games with weak interaction. On the theoretical side, we give extensive and rigorous existence and approximation properties of the graphon mean field solution in sufficiently large systems. On the practical side, we provide general learning schemes for graphon mean field equilibria by either introducing agent equivalence classes or reformulating the graphon mean field system as a classical mean field system. By repeatedly finding a regularized optimal control solution and its generated mean field, we successfully obtain plausible approximate Nash equilibria in otherwise infeasible large dense graph games with many agents. Empirically, we are able to demonstrate on a number of examples that the finite-agent behavior comes increasingly close to the mean field behavior for our computed equilibria as the graph or system size grows, verifying our theory. More generally, we successfully apply policy gradient reinforcement learning in conjunction with sequential Monte Carlo methods.
Computer Science and Game Theory,Machine Learning,Multiagent Systems,Optimization and Control
What problem does this paper attempt to address?
This paper attempts to solve the problem of finding approximate Nash equilibria in multi - agent systems on large - scale dense graphs. Specifically, the authors propose a discrete - time graph - theoretic mean - field game (Graphon Mean Field Games, GMFG) framework based on graphon theory to handle dynamic sequential games with a large number of agents. The main contributions of the paper include: 1. **Proposing a new discrete - time graph - theoretic mean - field game framework**: This is the first proposed general discrete - time graph - theoretic mean - field game framework, which is applicable to many problems that are essentially discrete - time or need to be controlled at discrete decision - making times. 2. **Providing theoretical analysis of the existence and approximation properties of the system**: The authors analyze in detail the existence and approximation properties of the graph - theoretic mean - field solution in a sufficiently large system, providing a theoretical basis for the effectiveness of the algorithm. 3. **Providing a general graph - theoretic mean - field equilibrium learning scheme**: The authors propose two methods to find the graph - theoretic mean - field equilibrium: - **Equivalence class method**: By introducing agent equivalence classes, the continuous interval \(I\) is divided into multiple subsets, and agents within each subset share the same dynamic characteristics. This method is suitable for handling general graph - theoretic problems and can handle an uncountable number of classes. - **Direct reinforcement learning method**: The graph - theoretic mean - field game is regarded as an extended state - space problem of the classical mean - field game, and the optimal strategy is directly solved using reinforcement learning techniques. 4. **Empirical evaluation**: The authors verify the proposed theoretical results through multiple experiments, showing that in a finite - agent graph system, as the graph or system size increases, the behavior of finite agents gradually approaches the mean - field behavior, thus finding a reasonable approximate Nash equilibrium. Overall, by combining graphon theory and mean - field game theory, this paper provides a new method to handle the Nash equilibrium problem in multi - agent systems on large - scale dense graphs, which has important theoretical and practical application values.