Abstract:Background: Digital health programs provide individualized support to patients with chronic diseases and their effectiveness is measured by the extent to which patients achieve target individual clinical outcomes and the program's ability to sustain patient engagement. However, patient dropout and inequitable intervention delivery strategies, which may unintentionally penalize certain patient subgroups, represent challenges to maximizing effectiveness. Therefore, methodologies that optimize the balance between success factors (achievement of target clinical outcomes and sustained engagement) equitably would be desirable, particularly when there are resource constraints. Objective: Our objectives were to propose a model for digital health program resource management that accounts jointly for the interaction between individual clinical outcomes and patient engagement, ensures equitable allocation as well as allows for capacity planning, and conducts extensive simulations using publicly available data on type 2 diabetes, a chronic disease. Methods: We propose a restless multiarmed bandit (RMAB) model to plan interventions that jointly optimize long-term engagement and individual clinical outcomes (in this case measured as the achievement of target healthy glucose levels). To mitigate the tendency of RMAB to achieve good aggregate performance by exacerbating disparities between groups, we propose new equitable objectives for RMAB and apply bilevel optimization algorithms to solve them. We formulated a model for the joint evolution of patient engagement and individual clinical outcome trajectory to capture the key dynamics of interest in digital chronic disease management programs. Results: In simulation exercises, our optimized intervention policies lead to up to 10% more patients reaching healthy glucose levels after 12 months, with a 10% reduction in dropout compared to standard-of-care baselines. Further, our new equitable policies reduce the mean absolute difference of engagement and health outcomes across 6 demographic groups by up to 85% compared to the state-of-the-art. Conclusions: Planning digital health interventions with individual clinical outcome objectives and long-term engagement dynamics as considerations can be both feasible and effective. We propose using an RMAB sequential decision-making framework, which may offer additional capabilities in capacity planning as well. The integration of an equitable RMAB algorithm further enhances the potential for reaching equitable solutions. This approach provides program designers with the flexibility to switch between different priorities and balance trade-offs across various objectives according to their preferences.

Modeling Mobile Health Users as Reinforcement Learning Agents

Optimizing Adaptive Notifications in Mobile Health Interventions Systems: Reinforcement Learning from a Data-driven Behavioral Simulator

Dyadic Reinforcement Learning

Group-driven Reinforcement Learning for Personalized mHealth Intervention.

A Robust Mixed-Effects Bandit Algorithm for Assessing Mobile Health Interventions

Reinforcement Learning Interventions on Boundedly Rational Human Agents in Frictionful Tasks

New Approach to Equitable Intervention Planning to Improve Engagement and Outcomes in a Digital Health Program: Simulation Study

Adapting Gamification Strategies According to Targeted Health Behavior Change: Findings from a Focus Group

Sufficient Markov Decision Processes with Alternating Deep Neural Networks

Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health

A reinforcement learning based algorithm for personalization of digital, just-in-time, adaptive interventions

Multi-Stage Receptivity Model for Mobile Just-In-Time Health Intervention

Cohesion-driven Online Actor-Critic Reinforcement Learning for Mhealth Intervention

Personalized Policy Learning Using Longitudinal Mobile Health Data

Challenges for Reinforcement Learning in Healthcare

Modeling user interaction with app-based reward system: A graphical model approach integrated with max-margin learning

Empowering Patients Using Smart Mobile Health Platforms: Evidence From A Randomized Field Experiment

Electronic health records based reinforcement learning for treatment optimizing

Offline Contextual Multi-armed Bandits for Mobile Health Interventions: A Case Study on Emotion Regulation

Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity

An Adaptive Agent-Based Interface for Personalized Health Interventions