Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment

Aamer Abdul Rahman,Pranav Agarwal,Rita Noumeir,Philippe Jouvet,Vincent Michalski,Samira Ebrahimi Kahou

2024-07-28

Abstract:Offline reinforcement learning has shown promise for solving tasks in safety-critical settings, such as clinical decision support. Its application, however, has been limited by the lack of interpretability and interactivity for clinicians. To address these challenges, we propose the medical decision transformer (MeDT), a novel and versatile framework based on the goal-conditioned reinforcement learning paradigm for sepsis treatment recommendation. MeDT uses the decision transformer architecture to learn a policy for drug dosage recommendation. During offline training, MeDT utilizes collected treatment trajectories to predict administered treatments for each time step, incorporating known treatment outcomes, target acuity scores, past treatment decisions, and current and past medical states. This analysis enables MeDT to capture complex dependencies among a patient's medical history, treatment decisions, outcomes, and short-term effects on stability. Our proposed conditioning uses acuity scores to address sparse reward issues and to facilitate clinician-model interactions, enhancing decision-making. Following training, MeDT can generate tailored treatment recommendations by conditioning on the desired positive outcome (survival) and user-specified short-term stability improvements. We carry out rigorous experiments on data from the MIMIC-III dataset and use off-policy evaluation to demonstrate that MeDT recommends interventions that outperform or are competitive with existing offline reinforcement learning methods while enabling a more interpretable, personalized and clinician-directed approach.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the issue of clinical decision support for sepsis patients in the Intensive Care Unit (ICU), specifically by developing a reinforcement learning-based method to recommend optimal drug dosages. Specifically, the goals of the paper are:

1. **Propose the Medical Decision Transformer (MeDT)**: This is a novel framework based on the goal-conditioned reinforcement learning paradigm for sepsis treatment recommendations. MeDT leverages the decision transformer architecture to learn a policy for recommending drug dosages.

2. **Address interpretability and interactivity issues in existing methods**: Current offline reinforcement learning methods have limitations in terms of interpretability and interaction with clinicians. MeDT addresses these issues by incorporating collected treatment trajectories during training to predict medication administration at each time step, considering known treatment outcomes, target acute scores, past treatment decisions, and current and past medical states.

3. **Handle the sparse reward problem**: By using acute scores as conditions, MeDT addresses the common sparse reward problem in reinforcement learning, which helps facilitate interaction between clinicians and the model and enhances decision-making.

4. **Achieve a personalized and clinically guided approach**: After training, MeDT can generate customized treatment recommendations based on desired positive outcomes (survival) and user-specified short-term stability improvements.

5. **Evaluate performance**: Through rigorous experiments on the MIMIC-III dataset and using off-policy evaluation (OPE) methods such as Fitted Q-Evaluation (FQE), Weighted Importance Sampling (WIS), and Weighted Doubly Robust (WDR), it is demonstrated that the interventions recommended by MeDT outperform or are comparable to existing offline reinforcement learning methods, while providing a more interpretable, personalized, and clinically guided approach.

In summary, this research aims to develop a system that better assists clinicians in making decisions when treating sepsis patients by combining clinical expertise and machine learning techniques, with the goal of improving treatment outcomes and patient survival rates.