Abstract:Background: Despite a great deal of interest in the application of artificial intelligence (AI) to sepsis/critical illness, most current approaches are limited in their potential impact: prediction models do not (and cannot) address the lack of effective therapeutics and current approaches to enhancing the treatment of sepsis focus on optimizing the application of existing interventions, and thus cannot address the development of new treatment options/modalities. The inability to test new therapeutic applications was highlighted by the generally unsatisfactory results from drug repurposing efforts in COVID-19. Hypothesis: Addressing this challenge requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training the game-playing AIs. We have previously demonstrated the potential of this method in the context of bacterial sepsis in which the microbial infection is responsive to antibiotic therapy. The current work addresses the control problem of multi-modal, adaptive immunomodulation in the circumstance where there is no effective anti-pathogen therapy (e.g., in a novel viral pandemic or in the face of resistant microbes). Methods: This is a proof-of-concept study that determines the controllability of sepsis without the ability to pharmacologically suppress the pathogen. We use as a surrogate system a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. The DRL algorithm 'trains' an AI on simulations of infection where both the control and observation spaces are limited to operating upon the defined immune mediators included in the IIRABM (a total of 11). Policies were learned using the Deep Deterministic Policy Gradient approach, with the objective function being a return to baseline system health. Results: DRL trained an AI policy that improved system mortality from 85% to 10.4%. Control actions affected every one of the 11 targetable cytokines and could be divided into those with static/unchanging controls and those with variable/adaptive controls. Adaptive controls primarily targeted 3 different aspects of the immune response: 2nd order pro-inflammation governing TH1/TH2 balance, primary anti-inflammation, and inflammatory cell proliferation. Discussion: The current treatment of sepsis is hampered by limitations in therapeutic options able to affect the biology of sepsis. This is heightened in circumstances where no effective antimicrobials exist, as was the case for COVID-19. Current AI methods are intrinsically unable to address this problem; doing so requires training AIs in contexts that fully represent the counterfactual space of potential treatments. The synthetic data needed for this task is only possible through the use of high-resolution, mechanism-based simulations. Finally, being able to treat sepsis will require a reorientation as to the sensing and actuating requirements needed to develop these simulations and bring them to the bedside.

Towards more efficient and robust evaluation of sepsis treatment with deep reinforcement learning

Reinforcement Learning with Balanced Clinical Reward for Sepsis Treatment

Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluation

Reinforcement Learning For Sepsis Treatment: A Continuous Action Space Solution

Deep reinforcement learning extracts the optimal sepsis treatment policy from treatment records

Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space

Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis Patients

Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach

Model-Based Reinforcement Learning for Sepsis Treatment

Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment

A value-based deep reinforcement learning model with human expertise in optimal treatment of sepsis

Clinical knowledge-guided deep reinforcement learning for sepsis antibiotic dosing recommendations

Artificial intelligence can use physiological parameters to optimize treatment strategies and predict clinical deterioration of sepsis in ICU

Optimal Treatment Strategies for Critical Patients with Deep Reinforcement Learning

Intelligent Medical Decision Making for Sepsis Detection using Reinforcement Learning

Offline reinforcement learning with uncertainty for treatment strategies in sepsis

Optimizing Sepsis Treatment Strategies Via a Reinforcement Learning Model

DeepAISE -- An End-to-End Development and Deployment of a Recurrent Neural Survival Model for Early Prediction of Sepsis

Dynamic Programming for Solving a Simulated Clinical Scenario of Sepsis Resuscitation

Optimal Sepsis Patient Treatment using Human-in-the-loop Artificial Intelligence

Preparing for the next COVID: Deep Reinforcement Learning trained Artificial Intelligence discovery of multi-modal immunomodulatory control of systemic inflammation in the absence of effective anti-microbials