Abstract:Self-play, where the algorithm learns by playing against itself without requiring any direct supervision, has become the new weapon in modern Reinforcement Learning (RL) for achieving superhuman performance in practice. However, the majority of exisiting theory in reinforcement learning only applies to the setting where the agent plays against a fixed environment; it remains largely open whether self-play algorithms can be provably effective, especially when it is necessary to manage the exploration/exploitation tradeoff. We study self-play in competitive reinforcement learning under the setting of Markov games, a generalization of Markov decision processes to the two-player case. We introduce a self-play algorithm---Value Iteration with Upper/Lower Confidence Bound (VI-ULCB)---and show that it achieves regret $\tilde{\mathcal{O}}(\sqrt{T})$ after playing $T$ steps of the game, where the regret is measured by the agent's performance against a \emph{fully adversarial} opponent who can exploit the agent's strategy at \emph{any} step. We also introduce an explore-then-exploit style algorithm, which achieves a slightly worse regret of $\tilde{\mathcal{O}}(T^{2/3})$, but is guaranteed to run in polynomial time even in the worst case. To the best of our knowledge, our work presents the first line of provably sample-efficient self-play algorithms for competitive reinforcement learning.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to design low - regret algorithms in Competitive Reinforcement Learning (CRL). Specifically, the paper focuses on self - play algorithms in two - player zero - sum Markov games. This type of game can be regarded as a special case of multi - player games, in which two players take actions simultaneously, but each player maximizes their own rewards, which leads to a conflict of interests between the players. The main contributions of the paper are as follows:
1. **Proposing the first sample - efficient self - play algorithm**: The paper introduces a self - play algorithm named Value Iteration with Upper/Lower Confidence Bound (VI - ULCB). This algorithm can achieve a regret of \(\tilde{O}(\sqrt{T})\) after \(T\) steps of the game. The regret measures the gap between the algorithm's performance and the strategy of a fully adversarial opponent. In addition, the paper also proposes an exploration - exploitation - style algorithm. Although its regret is slightly worse, \(\tilde{O}(T^{2/3})\), it can also guarantee polynomial - time complexity in the worst - case scenario.
2. **Theoretical analysis**: The paper provides a strict theoretical analysis, proves the effectiveness of the VI - ULCB algorithm, and discusses the performance of the algorithm under different settings. For example, in round - robin games, the regret of the VI - ULCB algorithm can be further optimized to \(\tilde{O}(\sqrt{H^3S^2(A + B)T})\) and has polynomial - time complexity.
3. **Balancing exploration and exploitation**: The paper deals with the trade - off between exploration and exploitation by introducing optimistic estimates. In zero - sum games, because the goals of the two players conflict with each other, the algorithm needs to consider both upper - bound and lower - bound estimates simultaneously to ensure that each player can take optimal actions according to their own goals.
4. **Computational efficiency**: Although the VI - ULCB algorithm performs well in terms of sample efficiency, it faces challenges in computational complexity. To this end, the paper also proposes a computationally efficient algorithm - Value Iteration after Exploration (VI - Explore), which runs in polynomial time with a regret of \(\tilde{O}(T^{2/3})\).
### Formula summary
- **Regret**:
\[
\text{Regret}(K)=\sum_{k = 1}^K\left[V^{\dagger},\nu_k(s_1^k)-V^{\mu_k,\dagger}(s_1^k)\right]
\]
- **Value function**:
\[
V^{\mu,\nu}_h(s)=\mathbb{E}_{\mu,\nu}\left[\sum_{h'=h}^H r_{h'}(s_{h'},a_{h'},b_{h'})\mid s_h = s\right]
\]
- **Q - value function**:
\[
Q^{\mu,\nu}_h(s,a,b)=\mathbb{E}_{\mu,\nu}\left[\sum_{h'=h}^H r_{h'}(s_{h'},a_{h'},b_{h'})\mid s_h = s,a_h = a,b_h = b\right]
\]
- **Bellman equation**:
\[
Q^{\mu,\nu}_h(s,a,b)=r_h(s,a,b)+P_hV^{\mu,\nu}_{h + 1}(s,a,b)
\]
\[
V^{\mu,\nu}_h(s)=\sum_{a,b}