Abstract:Public health practitioners often have the goal of monitoring patients and maximizing patients' time spent in "favorable" or healthy states while being constrained to using limited resources. Restless multi-armed bandits (RMAB) are an effective model to solve this problem as they are helpful to allocate limited resources among many agents under resource constraints, where patients behave differently depending on whether they are intervened on or not. However, RMABs assume the reward function is known. This is unrealistic in many public health settings because patients face unique challenges and it is impossible for a human to know who is most deserving of any intervention at such a large scale. To address this shortcoming, this paper is the first to present the use of inverse reinforcement learning (IRL) to learn desired rewards for RMABs, and we demonstrate improved outcomes in a maternal and child health telehealth program. First we allow public health experts to specify their goals at an aggregate or population level and propose an algorithm to design expert trajectories at scale based on those goals. Second, our algorithm WHIRL uses gradient updates to optimize the objective, allowing for efficient and accurate learning of RMAB rewards. Third, we compare with existing baselines and outperform those in terms of run-time and accuracy. Finally, we evaluate and show the usefulness of WHIRL on thousands on beneficiaries from a real-world maternal and child health setting in India. We publicly release our code here: <a class="link-external link-https" href="https://github.com/Gjain234/WHIRL" rel="external noopener nofollow">this https URL</a>.

Scalable Decision-Focused Learning in Restless Multi-Armed Bandits with Application to Maternal and Child Health

IRL for Restless Multi-Armed Bandits with Applications in Maternal and Child Health

A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health

Equitable Restless Multi-Armed Bandits: A General Framework Inspired By Digital Health

Collapsing Bandits and Their Application to Public Health Interventions

Learn to Intervene: An Adaptive Learning Policy for Restless Bandits in Application to Preventive Healthcare

Efficient Resource Allocation with Fairness Constraints in Restless Multi-Armed Bandits

Efficient Algorithms for Finite Horizon and Streaming Restless Multi-Armed Bandit Problems

Limited Resource Allocation in a Non-Markovian World: The Case of Maternal and Child Healthcare

The Bandit Whisperer: Communication Learning for Restless Bandits

Provably Efficient Reinforcement Learning for Adversarial Restless Multi-Armed Bandits with Unknown Transitions and Bandit Feedback

Planning and Learning in Risk-Aware Restless Multi-Arm Bandit Problem

A Bayesian Approach to Online Learning for Contextual Restless Bandits with Applications to Public Health

Efficient Public Health Intervention Planning Using Decomposition-Based Decision-Focused Learning

Simulation Based Algorithms for Markov Decision Processes and Multi-Action Restless Bandits

Optimizing Vital Sign Monitoring in Resource-Constrained Maternal Care: An RL-Based Restless Bandit Approach

A Federated Online Restless Bandit Framework for Cooperative Resource Allocation

A Risk-Averse Framework for Non-Stationary Stochastic Multi-Armed Bandits

Online Restless Multi-Armed Bandits with Long-Term Fairness Constraints

Non-stationary Bandits with Habituation and Recovery Dynamics and Knapsack Constraints