Shuang Qiu,Boxiang Lyu,Qinglin Meng,Zhaoran Wang,Zhuoran Yang,Michael I. Jordan
Abstract:Dynamic mechanism design studies how mechanism designers should allocate resources among agents in a time-varying environment. We consider the problem where the agents interact with the mechanism designer according to an unknown Markov Decision Process (MDP), where agent rewards and the mechanism designer's state evolve according to an episodic MDP with unknown reward functions and transition kernels. We focus on the online setting with linear function approximation and propose novel learning algorithms to recover the dynamic Vickrey-Clarke-Grove (VCG) mechanism over multiple rounds of interaction. A key contribution of our approach is incorporating reward-free online Reinforcement Learning (RL) to aid exploration over a rich policy space to estimate prices in the dynamic VCG mechanism. We show that the regret of our proposed method is upper bounded by $\tilde{\mathcal{O}}(T^{2/3})$ and further devise a lower bound to show that our algorithm is efficient, incurring the same $\tilde{\mathcal{O}}(T^{2 / 3})$ regret as the lower bound, where $T$ is the total number of rounds. Our work establishes the regret guarantee for online RL in solving dynamic mechanism design problems without prior knowledge of the underlying model.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to solve the problem of learning dynamic mechanism design in an unknown environment. Specifically, the authors study how to design resource allocation mechanisms in a time - varying environment, so that the mechanism designer (such as the seller) can interact with multiple agents (such as buyers) according to an unknown Markov decision process (MDP). The rewards of these agents and the state of the mechanism designer all evolve according to a piecewise - linear MDP with an unknown reward function and transition kernel.
#### Main challenges
1. **Unknown environment**: Existing research usually requires prior knowledge of key parameters or functions in the problem, such as the optimal strategy or the agents' valuation of goods. However, in real life, this information is often unknowable.
2. **Learning of dynamic VCG mechanisms**: The classic Vickrey - Clarke - Groves (VCG) mechanism is an effective mechanism design method for static environments. But in dynamic environments, the estimation of VCG prices becomes complicated because virtual policies in the absence of agents need to be considered, and these policies have never been executed.
#### Solutions
To address these challenges, the authors propose a new online reinforcement learning algorithm that can learn the dynamic VCG mechanism through multiple interactions without prior knowledge. Specifically:
1. **Exploration and exploitation**: The algorithm is divided into two stages:
- **Exploration stage**: Learn the underlying model through non - rewarding exploration to ensure sufficient coverage of all possible policy spaces, thereby reducing uncertainty.
- **Exploitation stage**: Solve the planning problem using the collected data set and execute a data - driven policy.
2. **Linear function approximation**: To solve the problem of large - scale state spaces, the algorithm introduces linear function approximation techniques.
3. **Performance guarantees**: The authors prove that this algorithm can achieve sub - linear regret upper bounds in terms of social welfare, agent utility, and seller utility, and these regret upper bounds are nearly optimal (i.e., minimax optimal).
4. **Mechanism design objectives**: The algorithm is also proven to approximately satisfy the three key requirements of mechanism design - truthfulness, individual rationality, and efficiency.
#### Application scenarios
This algorithm can be applied to a variety of practical problems, such as:
- **Dynamic sponsored search auctions**: The budgets and advertising values of advertisers change over time, and the algorithm can help optimize the allocation of advertising spaces.
- **Platform - as - a - Service (PaaS)**: Dynamic allocation of computing resources to balance power costs and user satisfaction.
In conclusion, this paper proposes a novel reinforcement learning method that can learn dynamic mechanism design in an unknown environment, solves the problem of relying on prior knowledge in existing methods, and provides theoretical performance guarantees.