Abstract:We introduce a stochastic principal-agent model. A principal and an agent interact in a stochastic environment, each privy to observations about the state not available to the other. The principal has the power of commitment, both to elicit information from the agent and to provide signals about her own information. The players communicate with each other and then select actions independently. Each of them receives a payoff based on the state and their joint action, and the environment transitions to a new state. The interaction continues over a finite time horizon. Both players are far-sighted, aiming to maximize their total payoffs over the time horizon. The model encompasses as special cases extensive-form games (EFGs) and stochastic games of incomplete information, partially observable Markov decision processes (POMDPs), as well as other forms of sequential principal-agent interactions, including Bayesian persuasion and automated mechanism design problems. We consider both the computation and learning of the principal's optimal policy. Since the general problem, which subsumes POMDPs, is intractable, we explore algorithmic solutions under hindsight observability, where the state and the interaction history are revealed at the end of each step. Though the problem becomes more amenable under this condition, the number of possible histories remains exponential in the length of the time horizon, making approaches for EFG-based models infeasible. We present an efficient algorithm based on the inducible value sets. The algorithm computes an $\epsilon$-approximate optimal policy in time polynomial in $1/\epsilon$. Additionally, we show an efficient learning algorithm for an episodic reinforcement learning setting where the transition probabilities are unknown. The algorithm guarantees sublinear regret $\tilde{O}(T^{2/3})$ for both players over $T$ episodes.

An Algorithm for Computing Stochastically Stable Distributions with Applications to Multiagent Learning in Repeated Games

Adaptive algorithm for multi-agent learning optimal cooperative pursuit strategy based on Markov game

On the Stability of Learning in Network Games with Many Players

A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning

Stochastic Principal-Agent Problems: Efficient Computation and Learning

Learning in Multi-Player Stochastic Games

Efficient Distributed Learning in Stochastic Non-cooperative Games without Information Exchange.

Distributed Multi-Coalition Games with General Linear Systems over Markovian Switching Networks

Decentralized learning in Markov games

Learning in Multi-level Stochastic games with Delayed Information

Stability of the Nash Equilibrium under Gradient Ascent Learning Algorithms in Two-Agent Two-Action Games

A unified stochastic approximation framework for learning in games

Distributionally robust stability of payoff allocations in stochastic coalitional games

Non-cooperative Multi-agent Systems with Exploring Agents

Stochastic Stability Analysis of Evolutionary Two-Player Games on Regular Graphs

Independent Learning in Stochastic Games

A Stochastic Game Framework for Analyzing Computational Investment Strategies in Distributed Computing

Stability in Random Hedonic Games

Computing Nash Equilibria in Multiplayer DAG-Structured Stochastic Games with Persistent Imperfect Information

A Polynomial-time Nash Equilibrium Algorithm for Repeated Stochastic Games

A Risk-Averse Equilibrium for Multi-Agent Systems