Abstract:We consider discrete-time stationary mean field games (MFG) with unknown dynamics and design algorithms for finding the equilibrium in finite time. Prior solutions to the problem build upon either the contraction assumption on a mean field optimality-consistency operator or strict weak monotonicity, which may be overly restrictive. In this work, we introduce a new class of solvable MFGs, "fully herding class", which expands the known finite-time solvable class of MFGs and for the first time includes problems with multiple equilibria. We propose a direct policy optimization method, Accelerated Single-loop Actor Critic Algorithm for Mean Field Games (ASAC-MFG), that provably finds a global equilibrium for MFGs within this class, under suitable access to a single trajectory of Markovian samples. Different from the prior arts, ASAC-MFG is single-loop and single-sample-path. We establish the finite-time and finite-sample convergence of ASAC-MFG to a mean field equilibrium via new techniques we develop for multi-time-scale stochastic approximation. We support the theoretical results with illustrative numerical simulations. When the mean field does not affect the transition and reward, a MFG reduces to a Markov decision process (MDP) and ASAC-MFG becomes an actor-critic algorithm for finding the optimal policy in average-reward MDPs, with a sample complexity matching the state-of-the-art. Existing works derive the complexity assuming a contraction on the Bellman operator, which is invalid for average-reward MDPs. We are able to match the rate while removing the untenable assumption through an improved Lyapunov function.

MF-OMO: An Optimization Formulation of Mean-Field Games

MF-OMO: An Optimization Formulation of Mean-Field Games

MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games

Linear-quadratic zero-sum mean-field type games: Optimality conditions and policy optimization

From Nash Equilibrium to Social Optimum and vice versa: a Mean Field Perspective

Discrete-time indefinite linear-quadratic mean field games and control: The finite-population case

A Single Online Agent Can Efficiently Learn Mean Field Games

Learning in Herding Mean Field Games: Single-Loop Algorithm with Finite-Time Convergence Analysis

On a Differential Generalized Nash Equilibrium Problem with Mean Field Interaction

A bilevel optimization method for inverse mean-field games

Social optimum of finite mean field games: existence and uniqueness of equilibrium solutions in the finite horizon and stationary solutions in the infinite horizon

Mixed Social Optima and Nash equilibrium in Linear-Quadratic-Gaussian Mean-field System

A General Framework for Learning Mean-Field Games

Mean-field games among teams

Direct Approach of Indefinite Linear-Quadratic Mean Field Games

Fictitious Play via Finite Differences for Mean Field Games with Optimal Stopping

Decomposition and Mean-Field Approach to Mixed Integer Optimal Compensation Problems

High order computation of optimal transport, mean field planning, and mean field games

Linear quadratic mean-field game-team analysis: A mixed coalition approach

Semi-Explicit Solutions to some Non-Linear Non-Quadratic Mean-Field-Type Games: A Direct Method