Abstract:We introduce a stochastic principal-agent model. A principal and an agent interact in a stochastic environment, each privy to observations about the state not available to the other. The principal has the power of commitment, both to elicit information from the agent and to provide signals about her own information. The players communicate with each other and then select actions independently. Each of them receives a payoff based on the state and their joint action, and the environment transitions to a new state. The interaction continues over a finite time horizon. Both players are far-sighted, aiming to maximize their total payoffs over the time horizon. The model encompasses as special cases extensive-form games (EFGs) and stochastic games of incomplete information, partially observable Markov decision processes (POMDPs), as well as other forms of sequential principal-agent interactions, including Bayesian persuasion and automated mechanism design problems. We consider both the computation and learning of the principal's optimal policy. Since the general problem, which subsumes POMDPs, is intractable, we explore algorithmic solutions under hindsight observability, where the state and the interaction history are revealed at the end of each step. Though the problem becomes more amenable under this condition, the number of possible histories remains exponential in the length of the time horizon, making approaches for EFG-based models infeasible. We present an efficient algorithm based on the inducible value sets. The algorithm computes an $\epsilon$-approximate optimal policy in time polynomial in $1/\epsilon$. Additionally, we show an efficient learning algorithm for an episodic reinforcement learning setting where the transition probabilities are unknown. The algorithm guarantees sublinear regret $\tilde{O}(T^{2/3})$ for both players over $T$ episodes.

Stochastic pairwise preference convergence in Bayesian agents

Stochastic Pairwise Preference Convergence in Bayesian Agents

Stochastic Principal-Agent Problems: Efficient Computation and Learning

Consensus Convergence with Stochastic Effects

A Stochastic Analysis of Particle Systems with Pairing

On Preference Learning Based on Sequential Bayesian Optimization with Pairwise Comparison

Expectations, Networks, and Conventions

Independent Learning in Stochastic Games

Interpersonal trust: Asymptotic analysis of a stochastic coordination game with multi-agent learning

Asymptotic Behavior of Bayesian Learners with Misspecified Models

Generalized replicator dynamics based on mean-field pairwise comparison dynamic

Stochastic Opinion Dynamics under Social Pressure in Arbitrary Networks

The Unification of Evolutionary Dynamics through the Bayesian Decay Factor in a Game on a Graph

Best-Response Dynamics in Lottery Contests

Updating Probabilities: A Complex Agent Based Example

Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks

Dynamics of measure-valued agents in the space of probabilities

Expectation Propagation with Stochastic Kinetic Model in Complex Interaction Systems.

Network Preference Dynamics using Lattice Theory

Beliefs in Decision-Making Cascades

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo