Abstract:Mean Field Control Games (MFCGs) provide a powerful theoretical framework for analyzing systems of infinitely many interacting agents, blending elements from Mean Field Games (MFGs) and Mean Field Control (MFC). However, solving the coupled Hamilton-Jacobi-Bellman and Fokker-Planck equations that characterize MFCG equilibria remains a significant computational challenge, particularly in high-dimensional or complex environments. This paper presents a scalable deep Reinforcement Learning (RL) approach to approximate equilibrium solutions of MFCGs. Building on previous works, We reformulate the infinite-agent stochastic control problem as a Markov Decision Process, where each representative agent interacts with the evolving mean field distribution. We use the actor-critic based algorithm from a previous paper (Angiuli <a class="link-external link-http" href="http://et.al" rel="external noopener nofollow">this http URL</a>., 2024) as the baseline and propose several versions of more scalable and efficient algorithms, utilizing techniques including parallel sample collection (batching); mini-batching; target network; proximal policy optimization (PPO); generalized advantage estimation (GAE); and entropy regularization. By leveraging these techniques, we effectively improved the efficiency, scalability, and training stability of the baseline algorithm. We evaluate our method on a linear-quadratic benchmark problem, where an analytical solution to the MFCG equilibrium is available. Our results show that some versions of our proposed approach achieve faster convergence and closely approximate the theoretical optimum, outperforming the baseline algorithm by an order of magnitude in sample efficiency. Our work lays the foundation for adapting deep RL to solve more complicated MFCGs closely related to real life, such as large-scale autonomous transportation systems, multi-firm economic competition, and inter-bank borrowing problems.

Model-Free Reinforcement Learning for Mean Field Games

Model-free Reinforcement Learning for Non-stationary Mean Field Games

Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL

Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces

Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning

Reinforcement Learning for Mean Field Game

Efficient and Scalable Deep Reinforcement Learning for Mean Field Control Games

Reinforcement Learning for Mean Field Games with Strategic Complementarities

Learning in Mean Field Games: A Survey

Agent-Level Maximum Entropy Inverse Reinforcement Learning for Mean Field Games

Reinforcement Learning for Finite Space Mean-Field Type Games

On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation

Scalable Offline Reinforcement Learning for Mean Field Games

Maximum Causal Entropy Inverse Reinforcement Learning for Mean-Field Games

When is Mean-Field Reinforcement Learning Tractable and Relevant?

Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective

Provable Fictitious Play for General Mean-Field Games

Unified Reinforcement Q-Learning for Mean Field Game and Control Problems

Meta-Inverse Reinforcement Learning for Mean Field Games Via Probabilistic Context Variables

Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games

A General Framework for Learning Mean-Field Games