Abstract:We consider discrete-time stationary mean field games (MFG) with unknown dynamics and design algorithms for finding the equilibrium in finite time. Prior solutions to the problem build upon either the contraction assumption on a mean field optimality-consistency operator or strict weak monotonicity, which may be overly restrictive. In this work, we introduce a new class of solvable MFGs, "fully herding class", which expands the known finite-time solvable class of MFGs and for the first time includes problems with multiple equilibria. We propose a direct policy optimization method, Accelerated Single-loop Actor Critic Algorithm for Mean Field Games (ASAC-MFG), that provably finds a global equilibrium for MFGs within this class, under suitable access to a single trajectory of Markovian samples. Different from the prior arts, ASAC-MFG is single-loop and single-sample-path. We establish the finite-time and finite-sample convergence of ASAC-MFG to a mean field equilibrium via new techniques we develop for multi-time-scale stochastic approximation. We support the theoretical results with illustrative numerical simulations. When the mean field does not affect the transition and reward, a MFG reduces to a Markov decision process (MDP) and ASAC-MFG becomes an actor-critic algorithm for finding the optimal policy in average-reward MDPs, with a sample complexity matching the state-of-the-art. Existing works derive the complexity assuming a contraction on the Bellman operator, which is invalid for average-reward MDPs. We are able to match the rate while removing the untenable assumption through an improved Lyapunov function.

An Infinite-Horizon Mean Field Game of Growth and Capital Accumulation: A Markov Chain Approximation Numerical Scheme and Its Challenges

A Probabilistic Approach to Discounted Infinite Horizon and Invariant Mean Field Games

Introduction to Mean Field Games and Mean-Field-Type Games

A Mean Field Capital Accumulation Game with HARA Utility

Approximation of deterministic mean field games under polynomial growth conditions on the data

A hybrid deep learning method for finite-horizon mean-field game problems

A mean-field game economic growth model

Multiple-Population Discrete-Time Mean Field Games with Discounted and Total Payoffs: Approximation of Games with Finite Populations

Learning Graphon Mean Field Games and Approximate Nash Equilibria

Direct Approach of Indefinite Linear-Quadratic Mean Field Games

Asymptotic Nash Equilibria of Finite-State Ergodic Markovian Mean Field Games

Mean Field Asymptotics of Markov Decision Evolutionary Games and Teams

Mean field game of optimal relative investment with jump risk

Ε-Nash Mean-Field Games for Linear-Quadratic Systems with Random Jumps and Applications.

Existence and uniqueness results for a mean-field game of optimal investment

Optimal investment in a large population of competitive and heterogeneous agents

Backward Linear-Quadratic Mean Field Stochastic Differential Games: A Direct Method

Continuous-time mean field Markov decision models

Robustness and Approximation of Discrete-time Mean-field Games under Discounted Cost Criterion

Learning in Herding Mean Field Games: Single-Loop Algorithm with Finite-Time Convergence Analysis

A Mean Field Game Approach to Relative Investment-Consumption Games with Habit Formation