Abstract:Background: Reinforcement learning (RL) holds great promise for intensive care medicine given the abundant availability of data and frequent sequential decision-making. But despite the emergence of promising algorithms, RL driven bedside clinical decision support is still far from reality. Major challenges include trust and safety. To help address these issues, we introduce cross off-policy evaluation and policy restriction and show how detailed policy analysis may increase clinical interpretability. As an example, we apply these in the setting of RL to optimise ventilator settings in intubated covid-19 patients. Methods: With data from the Dutch ICU Data Warehouse and using an exhaustive hyperparameter grid search, we identified an optimal set of Dueling Double-Deep Q Network RL models. The state space comprised ventilator, medication, and clinical data. The action space focused on positive end-expiratory pressure (peep) and fraction of inspired oxygen (FiO2) concentration. We used gas exchange indices as interim rewards, and mortality and state duration as final rewards. We designed a novel evaluation method called cross off-policy evaluation (OPE) to assess the efficacy of models under varying weightings between the interim and terminal reward components. In addition, we implemented policy restriction to prevent potentially hazardous model actions. We introduce delta-Q to compare physician versus policy action quality and in-depth policy inspection using visualisations. Results: We created trajectories for 1118 intensive care unit (ICU) admissions and trained 69,120 models using 8 model architectures with 128 hyperparameter combinations. For each model, policy restrictions were applied. In the first evaluation step, 17,182/138,240 policies had good performance, but cross-OPE revealed suboptimal performance for 44% of those by varying the reward function used for evaluation. Clinical policy inspection facilitated assessment of action decisions for individual patients, including identification of action space regions that may benefit most from optimisation. Conclusion: Cross-OPE can serve as a robust evaluation framework for safe RL model implementation by identifying policies with good generalisability. Policy restriction helps prevent potentially unsafe model recommendations. Finally, the novel delta-Q metric can be used to operationalise RL models in clinical practice. Our findings offer a promising pathway towards application of RL in intensive care medicine and beyond.

Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care

Optimal Treatment Strategies for Critical Patients with Deep Reinforcement Learning

Reinforcement learning for intensive care medicine: actionable clinical insights from novel approaches to reward shaping and off-policy model evaluation

Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications

Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis Patients

Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluation

Learning medical triage from clinicians using Deep Q-Learning

Pruning the Path to Optimal Care: Identifying Systematically Suboptimal Medical Decision-Making with Inverse Reinforcement Learning

Deep Reinforcement Learning for Cost-Effective Medical Diagnosis

Reinforcement Learning For Survival, A Clinically Motivated Method For Critically Ill Patients

Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning

Reinforcement Learning in Clinical Medicine: a Method to Optimize Dynamic Treatment Regime over Time.

Balancing therapeutic effect and safety in ventilator parameter recommendation: An offline reinforcement learning approach

Dynamic Programming for Solving a Simulated Clinical Scenario of Sepsis Resuscitation

Adaptive Multi-Agent Deep Reinforcement Learning for Timely Healthcare Interventions

Medical Dead-ends and Learning to Identify High-risk States and Treatments

Reinforcement Learning with Balanced Clinical Reward for Sepsis Treatment

Reinforcement Learning for Clinical Decision Support in Critical Care: Comprehensive Review

Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare

Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination

Deep Reinforcement Learning for Efficient and Fair Allocation of Health Care Resources