Abstract:We consider finite-horizon restless bandits with multiple pulls per period, which play an important role in recommender systems, active learning, revenue management, and many other areas. While an optimal policy can be computed, in principle, using dynamic programming, the computation required scales exponentially in the number of arms $N$. Thus, there is substantial value in understanding the performance of index policies and other policies that can be computed efficiently for large $N$. We study the growth of the optimality gap, i.e., the loss in expected performance compared to an optimal policy, for such policies in a classical asymptotic regime proposed by Whittle in which $N$ grows while holding constant the fraction of arms that can be pulled per period. Intuition from the Central Limit Theorem and the tightest previous theoretical bounds suggest that this optimality gap should grow like $O(\sqrt{N})$. Surprisingly, we show that it is possible to outperform this bound. We characterize a non-degeneracy condition and a wide class of novel practically-computable policies, called fluid-priority policies, in which the optimality gap is $O(1)$. These include most widely-used index policies. When this non-degeneracy condition does not hold, we show that fluid-priority policies nevertheless have an optimality gap that is $O(\sqrt{N})$, significantly generalizing the class of policies for which convergence rates are known. We demonstrate that fluid-priority policies offer state-of-the-art performance on a collection of restless bandit problems in numerical experiments.

Restless Bandits with Average Reward: Breaking the Uniform Global Attractor Assumption

Achieving Exponential Asymptotic Optimality in Average-Reward Restless Bandits without Global Attractor Assumption

Unichain and Aperiodicity are Sufficient for Asymptotic Optimality of Average-Reward Restless Bandits

Near-optimality for infinite-horizon restless bandits with many arms

Restless Bandits with Many Arms: Beating the Central Limit Theorem

An Optimal-Control Approach to Infinite-Horizon Restless Bandits: Achieving Asymptotic Optimality with Minimal Assumptions

Restless-UCB, an Efficient and Low-complexity Algorithm for Online Restless Bandits.

Networked Restless Bandits with Positive Externalities

Non-stationary Bandits with Habituation and Recovery Dynamics and Knapsack Constraints

LP-based policies for restless bandits: necessary and sufficient conditions for (exponentially fast) asymptotic optimality

Achieving O(1/N) Optimality Gap in Restless Bandits through Diffusion Approximation

Low-Complexity Algorithm for Restless Bandits with Imperfect Observations

Congested Bandits: Optimal Routing via Short-term Resets

On Optimality of Greedy Policy for a Class of Standard Reward Function of Restless Multi-armed Bandit Problem

Model Predictive Control is Almost Optimal for Restless Bandit

Restless Linear Bandits

Exploration for Free: How Does Reward Heterogeneity Improve Regret in Cooperative Multi-agent Bandits?

The Non-Bayesian Restless Multi-Armed Bandit: A Case of Near-Logarithmic Strict Regret

Lagrangian Index Policy for Restless Bandits with Average Reward

Optimality of Myopic Policy for Restless Multiarmed Bandit with Imperfect Observation

Decentralized Stochastic Multi-Player Multi-Armed Walking Bandits