Abstract:Consider $N$ players each with a $d$-dimensional action set. Each of the players' utility functions includes their reward function and a linear term for each dimension, with coefficients that are controlled by the manager. We assume that the game is strongly monotone, so if each player runs gradient descent, the dynamics converge to a unique Nash equilibrium (NE). The NE is typically inefficient in terms of global performance. The resulting global performance of the system can be improved by imposing $K$-dimensional linear constraints on the NE. We therefore want the manager to pick the controlled coefficients that impose the desired constraint on the NE. However, this requires knowing the players' reward functions and their action sets. Obtaining this game structure information is infeasible in a large-scale network and violates the users' privacy. To overcome this, we propose a simple algorithm that learns to shift the NE of the game to meet the linear constraints by adjusting the controlled coefficients online. Our algorithm only requires the linear constraints violation as feedback and does not need to know the reward functions or the action sets. We prove that our algorithm, which is based on two time-scale stochastic approximation, guarantees convergence with probability 1 to the set of NE that meet target linear constraints. We then provide a mean square convergence rate of $O(t^{-1/4})$ for our algorithm. This is the first such bound for two time-scale stochastic approximation where the slower time-scale is a fixed point iteration with a non-expansive mapping. We demonstrate how our scheme can be applied to optimizing a global quadratic cost at NE and load balancing in resource allocation games. We provide simulations of our algorithm for these scenarios.

Newton’s Method, Bellman Recursion and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Adaptive Dynamic Programming for a Nonlinear Two‐Player Non‐Zero‐Sum Differential Game With State and Input Constraints

Residual Descent Differential Dynamic Game (RD3G) -- A Fast Newton Solver for Constrained General Sum Games

The computation of approximate feedback Stackelberg equilibria in multi-player nonlinear constrained dynamic games

Nash Equilibria for Linear Quadratic Discrete-time Dynamic Games via Iterative and Data-driven Algorithms

The Computation of Approximate Feedback Stackelberg Equilibria in Multiplayer Nonlinear Constrained Dynamic Games

Differential Variational Inequality Approach to Dynamic Games with Shared Constraints

A Policy Iteration Algorithm for N-player General-Sum Linear Quadratic Dynamic Games

Min–max adaptive dynamic programming for zero-sum differential games

A Sequential Quadratic Programming Approach to the Solution of Open-Loop Generalized Nash Equilibria for Autonomous Racing

Approximate dynamic programming for continuous state and control problems

Learning to Control Unknown Strongly Monotone Games

A Convex Optimization Approach to Dynamic Programming in Continuous State and Action Spaces

Dynamic Programming Approach to Discrete Time Dynamic Feedback Stackelberg Games with Independent and Dependent Followers.

Semi-Explicit Solutions to some Non-Linear Non-Quadratic Mean-Field-Type Games: A Direct Method

Reinforcement Learning for Inverse Linear-quadratic Dynamic Non-cooperative Games

Distributed-Observer-Based Nash Equilibrium Seeking Algorithm for Quadratic Games With Nonlinear Dynamics

On the Decomposition of Differential Game

Differential-game for resource aware approximate optimal control of large-scale nonlinear systems with multiple players

Recursive stochastic differential games with non-Lipschitzian generatorsand viscosity solutions of Hamilton-Jacobi-Bellman-Isaacs equations

Differential dynamic programming with stagewise equality and inequality constraints using interior point method