Abstract:We introduce a stochastic principal-agent model. A principal and an agent interact in a stochastic environment, each privy to observations about the state not available to the other. The principal has the power of commitment, both to elicit information from the agent and to provide signals about her own information. The players communicate with each other and then select actions independently. Each of them receives a payoff based on the state and their joint action, and the environment transitions to a new state. The interaction continues over a finite time horizon. Both players are far-sighted, aiming to maximize their total payoffs over the time horizon. The model encompasses as special cases extensive-form games (EFGs) and stochastic games of incomplete information, partially observable Markov decision processes (POMDPs), as well as other forms of sequential principal-agent interactions, including Bayesian persuasion and automated mechanism design problems. We consider both the computation and learning of the principal's optimal policy. Since the general problem, which subsumes POMDPs, is intractable, we explore algorithmic solutions under hindsight observability, where the state and the interaction history are revealed at the end of each step. Though the problem becomes more amenable under this condition, the number of possible histories remains exponential in the length of the time horizon, making approaches for EFG-based models infeasible. We present an efficient algorithm based on the inducible value sets. The algorithm computes an $\epsilon$-approximate optimal policy in time polynomial in $1/\epsilon$. Additionally, we show an efficient learning algorithm for an episodic reinforcement learning setting where the transition probabilities are unknown. The algorithm guarantees sublinear regret $\tilde{O}(T^{2/3})$ for both players over $T$ episodes.

On the complexity of computing Markov perfect equilibrium in general-sum stochastic games

Corrigendum to On the complexity of computing Markov perfect equilibrium in general-sum stochastic games

Computing perfect stationary equilibria in stochastic games

Stochastic Game with Interactive Information Acquisition: Pipelined Perfect Markov Bayesian Equilibrium

When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?

Computing Nash Equilibria in Multiplayer DAG-Structured Stochastic Games with Persistent Imperfect Information

Stationary Markov Perfect Equilibria in Discounted Stochastic Games

Distributed Multi-Coalition Games with General Linear Systems over Markovian Switching Networks

Computational Complexity of Computing a Quasi-Proper Equilibrium

Geometric Structure and Polynomial-time Algorithm of Game Equilibria

Tractable Equilibrium Computation in Markov Games through Risk Aversion

On the computational complexity of solving stochastic mean-payoff games

The Computational Complexity of Single-Player Imperfect-Recall Games

Robustness and Sample Complexity of Model-Based MARL for General-Sum Markov Games

Computing Nash Equilibria: Approximation and Smoothed Complexity

Imperfect-Recall Games: Equilibrium Concepts and Their Complexity

General sum stochastic games with networked information flows

Stationary Almost Markov ε-Equilibria for Discounted Stochastic Games with Borel Spaces and Unbounded Payoffs

20 06 Computing Nash Equilibria : Approximation and Smoothed Complexity

Stochastic Principal-Agent Problems: Efficient Computation and Learning

Identification and Estimation of Dynamic Games with Unknown Information Structure