A New Standard for Sepsis Prediction Algorithms: Using Time-Dependent Analysis for Earlier Clinically Relevant Alerts

Jenish Maharjan,Rahul Thapa,Jacob Calvert,Misty M Attwood,Sepideh Shokouhi,Satish Casie Chetty,Zohora Iqbal,Navan Singh,Rome Arnold,Jana Hoffman,Samson Mataraso,Anurag Garikipati,Carson Lam,Qingqing Mao

DOI: https://doi.org/10.2139/ssrn.4130480

2022-01-01

Abstract:Timely recognition of sepsis in hospital patients increases the likelihood of patient survival. The value of sepsis alert systems in clinical settings is diminished if these alerts are generated after clinically relevant times: after clinician suspicion of sepsis (clinical evaluation or treatment for sepsis) or onset of sepsis (defined by SOFA score). We evaluate and compare two models - standard and early - using traditional time-agnostic methods (area under the curve: AUC; true positive rate: TPR) and a time-dependent approach with redefined metrics that account for the timing of alerts; i.e., the clinical relevancy or earliness (eAUC and eTPR), for three different endpoints. When evaluating the models at the clinically relevant alert-before-sepsis-onset endpoint, the early model outperforms the standard model and achieves eAUC values on the holdout test and external validation sets of (0.892; 0.880) vs (0.832; 0.781), respectively. The early model generated more than 70% of the correct alerts within the clinically optimizable timeframe for preventive measures from 24 hours prior up to sepsis onset at 0.8 TPR. We demonstrate that time-agnostic evaluations of algorithm performance may not accurately represent the usefulness of the model in practice; while an algorithm may achieve high accuracy as measured by high AUC values, this does not describe the potential clinical relevance of the alerts.