Abstract:Restless multi-armed bandits (RMAB) have been widely used to model sequential decision making problems with constraints. The decision maker (DM) aims to maximize the expected total reward over an infinite horizon under an "instantaneous activation constraint" that at most B arms can be activated at any decision epoch, where the state of each arm evolves stochastically according to a Markov decision process (MDP). However, this basic model fails to provide any fairness guarantee among arms. In this paper, we introduce RMAB-F, a new RMAB model with "long-term fairness constraints", where the objective now is to maximize the long term reward while a minimum long-term activation fraction for each arm must be satisfied. For the online RMAB-F setting (i.e., the underlying MDPs associated with each arm are unknown to the DM), we develop a novel reinforcement learning (RL) algorithm named Fair-UCRL. We prove that Fair-UCRL ensures probabilistic sublinear bounds on both the reward regret and the fairness violation regret. Compared with off-the-shelf RL methods, our Fair-UCRL is much more computationally efficient since it contains a novel exploitation that leverages a low-complexity index policy for making decisions. Experimental results further demonstrate the effectiveness of our Fair-UCRL.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the Restless Multi - Armed Bandits (RMAB) problem with long - term fairness constraints. Specifically, it introduces a new model, RMAB - F, to ensure that while maximizing long - term rewards, the number of activations for each arm meets the minimum long - term activation ratio. #### Main problems: 1. **Limitations of the basic RMAB model**: - Although the basic RMAB model can handle sequential decision - making problems with constraints, it cannot provide any fairness guarantees regarding how to allocate activation opportunities among different arms. - This is an important design and ethical issue in practical applications, such as reducing data bias in healthcare and providing quality - of - service guarantees in network resource allocation. 2. **Introducing long - term fairness constraints**: - The paper proposes a new RMAB model, RMAB - F, in which "long - term fairness constraints" are added. That is, the decision - maker (DM) must ensure that the long - term activation ratio of each arm is not lower than a set minimum value. - The specific goal is to maximize the long - term total reward under the premise of satisfying the instantaneous activation limits in each decision cycle and the long - term fairness requirements of each arm. 3. **Challenges in the online environment**: - In the online environment, the DM does not know the MDP (Markov Decision Process) corresponding to each arm, which makes the problem more complex. - Existing RL algorithms (such as colored - UCRL2, Thompson sampling method) have problems of exponential growth in computational complexity and regret bounds, and do not consider fairness constraints. 4. **Developing an efficient reinforcement learning algorithm**: - To solve the above problems, the paper develops a new reinforcement learning algorithm, Fair - UCRL, which is not only computationally efficient but also can provide probability sub - linear bounds on reward regret and fairness - violation regret. #### Summary of mathematical formulas: - **Objective of maximizing long - term rewards**: \[ \max_{\pi} \liminf_{T \to \infty} \frac{1}{T} E\left[\sum_{t = 1}^{T} \sum_{n = 1}^{N} r_n(t)\right] \] - **Instantaneous activation constraints**: \[ \sum_{n = 1}^{N} a_n(t) \leq B, \quad \forall t \] - **Long - term fairness constraints**: \[ \liminf_{T \to \infty} \frac{1}{T} E\left[\sum_{t = 1}^{T} a_n(t)\right] \geq \eta_n, \quad \forall n \in N \] By introducing these constraints, the paper aims to develop a reinforcement learning algorithm that can ensure fairness as well as be computationally efficient, so as to better deal with resource allocation in the real world, healthcare and other problems.

Online Restless Multi-Armed Bandits with Long-Term Fairness Constraints

Fairness of Exposure in Online Restless Multi-armed Bandits

Efficient Resource Allocation with Fairness Constraints in Restless Multi-Armed Bandits

Provably Efficient Reinforcement Learning for Adversarial Restless Multi-Armed Bandits with Unknown Transitions and Bandit Feedback

A Federated Online Restless Bandit Framework for Cooperative Resource Allocation

DOPL: Direct Online Preference Learning for Restless Bandits with Preference Feedback

On Multi-Armed Bandit with Impatient Arms

Restless-UCB, an Efficient and Low-complexity Algorithm for Online Restless Bandits.

Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertainty

Equitable Restless Multi-Armed Bandits: A General Framework Inspired By Digital Health

The Bandit Whisperer: Communication Learning for Restless Bandits

Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedback

Auction-Based Combinatorial Multi-Armed Bandit Mechanisms with Strategic Arms

Reinforcement Learning Augmented Asymptotically Optimal Index Policy for Finite-Horizon Restless Bandits.

Competing for Shareable Arms in Multi-Player Multi-Armed Bandits

Multiarmed Bandits Problem Under the Mean-Variance Setting

Multi-Fidelity Multi-Armed Bandits Revisited

Uniform Last-Iterate Guarantee for Bandits and Reinforcement Learning

Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewards

Multi-Armed Bandits with Abstention

RL-CFR: Improving Action Abstraction for Imperfect Information Extensive-Form Games with Reinforcement Learning