Abstract:The stochastic multi-armed bandit model captures the tradeoff between exploration and exploitation. We study the effects of competition and cooperation on this tradeoff. Suppose there are $k$ arms and two players, Alice and Bob. In every round, each player pulls an arm, receives the resulting reward, and observes the choice of the other player but not their reward. Alice's utility is $\Gamma_A + \lambda \Gamma_B$ (and similarly for Bob), where $\Gamma_A$ is Alice's total reward and $\lambda \in [-1, 1]$ is a cooperation parameter. At $\lambda = -1$ the players are competing in a zero-sum game, at $\lambda = 1$, they are fully cooperating, and at $\lambda = 0$, they are neutral: each player's utility is their own reward. The model is related to the economics literature on strategic experimentation, where usually players observe each other's rewards. With discount factor $\beta$, the Gittins index reduces the one-player problem to the comparison between a risky arm, with a prior $\mu$, and a predictable arm, with success probability $p$. The value of $p$ where the player is indifferent between the arms is the Gittins index $g = g(\mu,\beta) > m$, where $m$ is the mean of the risky arm. We show that competing players explore less than a single player: there is $p^* \in (m, g)$ so that for all $p > p^*$, the players stay at the predictable arm. However, the players are not myopic: they still explore for some $p > m$. On the other hand, cooperating players explore more than a single player. We also show that neutral players learn from each other, receiving strictly higher total rewards than they would playing alone, for all $ p\in (p^*, g)$, where $p^*$ is the threshold from the competing case. Finally, we show that competing and neutral players eventually settle on the same arm in every Nash equilibrium, while this can fail for cooperating players.

Parallel Best Arm Identification in Heterogeneous Environments

Communication-Efficient Collaborative Best Arm Identification

Collaborative Learning with Limited Interaction: Tight Bounds for Distributed Exploration in Multi-armed Bandits

Collaborative Top Distribution Identifications with Limited Interaction (extended Abstract).

Multi-Agent Best Arm Identification in Stochastic Linear Bandits

Distributed Bandits with Heterogeneous Agents

Cost Aware Best Arm Identification

Collaboratively Learning the Best Option, Using Bounded Memory

Competing Bandits in Decentralized Large Contextual Matching Markets

Enhancing Collaboration in Heterogeneous Multiagent Systems Through Communication Complementary Graph

Best Arm Identification in Batched Multi-armed Bandit Problems

The Role of Contextual Information in Best Arm Identification

Matching in Multi-arm Bandit with Collision

Open Problem: Optimal Best Arm Identification with Fixed Budget

Best Arm Identification with Resource Constraints

Multiplayer Bandit Learning, from Competition to Cooperation

On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments

Identifying the Best Arm in the Presence of Global Environment Shifts

Best arm identification in multi-armed bandits with delayed feedback

Open Problem: Best Arm Identification: Almost Instance-Wise Optimality and the Gap Entropy Conjecture

Multi-agent Multi-armed Bandits with Stochastic Sharable Arm Capacities