Abstract:We design and analyze reinforcement learning algorithms for Graphon Mean-Field Games (GMFGs). In contrast to previous works that require the precise values of the graphons, we aim to learn the Nash Equilibrium (NE) of the regularized GMFGs when the graphons are unknown. Our contributions are threefold. First, we propose the Proximal Policy Optimization for GMFG (GMFG-PPO) algorithm and show that it converges at a rate of $O(T^{-1/3})$ after $T$ iterations with an estimation oracle, improving on a previous work by Xie et al. (ICML, 2021). Second, using kernel embedding of distributions, we design efficient algorithms to estimate the transition kernels, reward functions, and graphons from sampled agents. Convergence rates are then derived when the positions of the agents are either known or unknown. Results for the combination of the optimization algorithm GMFG-PPO and the estimation algorithm are then provided. These algorithms are the first specifically designed for learning graphons from sampled agents. Finally, the efficacy of the proposed algorithms are corroborated through simulations. These simulations demonstrate that learning the unknown graphons reduces the exploitability effectively.

What problem does this paper attempt to address?

This paper aims to solve the learning problem of Nash Equilibrium (NE) in Graphon Mean - Field Games (GMFGs), especially when the graphons are unknown. Specifically, the main contributions of the paper and the problems solved can be summarized as follows: 1. **Propose and analyze the GMFG - PPO algorithm**: - The paper designs a proximal policy optimization algorithm called GMFG - PPO for learning the Nash equilibrium in regularized GMFGs. - Given an estimator, this algorithm has a convergence rate of $\tilde{O}(T^{-1/3})$ after $T$ iterations, which is faster than $\tilde{O}(T^{-1/5})$ in previous works (such as Xie et al., 2021). - The algorithm overcomes optimization - related challenges through carefully designed policies and environmental update rates, and proves a faster convergence speed under fewer assumptions. 2. **Design an efficient model - learning algorithm**: - Using the method of kernel - embedded distributions, efficient algorithms are designed to estimate the transition kernel, reward function, and graphon from sampled agents. - Provide convergence rates when the positions of agents are known or unknown. - These algorithms are the first to be specifically designed for learning graphons from sampled agents. 3. **Summary of theoretical results**: - The paper provides multiple theorems and corollaries, which describe in detail the convergence rates of model estimation and Nash equilibrium learning algorithms under different sampling schemes. - For example, when the positions of agents are known and fixed, the convergence rate of model estimation is $O((NL)^{-1}+N^{-1/2})$, where $N$ is the number of sampled agents and $L$ is the number of samplings for each agent. 4. **Numerical simulation verification**: - The effectiveness of the proposed algorithms is verified by numerical simulations, indicating that learning unknown graphons can significantly reduce the exploitability. ### Formula Representation Some key formulas involved in the paper include: - **Definition of Nash equilibrium**: \[ J_{\lambda,\alpha}(\pi^*, \mu^*, W^*) \geq J_{\lambda,\alpha}(\tilde{\pi}_\alpha, \mu^*, W^*) \quad \forall \alpha \in I, \tilde{\pi}_\alpha \in \Pi^H \] \[ \mu^* = \Gamma_2(\pi^*, W^*) \] - **Definitions of value function and action - value function**: \[ V_{\lambda,\alpha}^h(s, \pi_\alpha, \mu^I, W^*) = \mathbb{E}_{\pi_\alpha} \left[ \sum_{t = h}^H \left( r_t^*(s_t^\alpha, a_t^\alpha, z_t^\alpha(\mu_t^I, W_t^*)) - \lambda \log \pi_t(a_t^\alpha|s_t^\alpha) \right) \mid s_h^\alpha = s \right] \] \[ Q_{\lambda,\alpha}^h(s, a, \pi_\alpha, \mu^I, W^*) = r_h^*(s, a, z_h^\alpha(\mu_h^I, W_h^*)) + \mathbb{E} \left[ V_{\lambda,\alpha}^{h + 1}(s_{h+1}^\alpha, \pi_\alpha, \mu^I, W^*) \mid s_h^\alpha = s, a_h^\alpha = a \right]

Learning Regularized Graphon Mean-Field Games with Unknown Graphons

Learning Regularized Monotone Graphon Mean-Field Games

Learning Mean Field Games on Sparse Graphs: A Hybrid Graphex Approach

Learning Graphon Mean Field Games and Approximate Nash Equilibria

A General Framework for Learning Mean-Field Games

Graphon Mean Field Games with a Representative Player: Analysis and Learning Algorithm

A Single Online Agent Can Efficiently Learn Mean Field Games

Last-iterate Convergence in Regularized Graphon Mean Field Game

Infinite horizon LQG Graphon Mean Field Games: Explicit Nash values and local minima

Provable Fictitious Play for General Mean-Field Games

Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation

Bounded Rationality Equilibrium Learning in Mean Field Games

Linear-Quadratic Graphon Mean Field Games with Common Noise

Reinforcement Learning for SBM Graphon Games with Re-Sampling

Regularization of the policy updates for stabilizing Mean Field Games

Deep Backward and Galerkin Methods for the Finite State Master Equation

Learning in Mean Field Games: A Survey

Representation Learning for General-sum Low-rank Markov Games

Graphon Mean-Field Control for Cooperative Multi-Agent Reinforcement Learning

Deep Learning for Mean Field Games with non-separable Hamiltonians

Numerical methods for mean field games based on Gaussian processes and Fourier features