Abstract:Importance: Risk-stratification tools are routinely used in obstetrics to assist care teams in assessing and communicating risk associated with delivery. Electronic health record data and machine learning methods may offer a novel opportunity to improve and automate risk assessment. Objective: To compare the predictive performance of natural language processing (NLP) of clinician documentation with that of a previously validated tool to identify individuals at high risk for maternal morbidity. Design, setting, and participants: This retrospective diagnostic study was conducted at Brigham and Women's Hospital and Massachusetts General Hospital, Boston, Massachusetts, and included individuals admitted for delivery at the former institution from July 1, 2016, to February 29, 2020. A subset of these encounters (admissions from February to December 2018) was part of a previous prospective validation study of the Obstetric Comorbidity Index (OB-CMI), a comorbidity-weighted score to stratify risk of severe maternal morbidity (SMM). Exposures: Natural language processing of clinician documentation and OB-CMI scores. Main outcomes and measures: Natural language processing of clinician-authored admission notes was used to predict SMM in individuals delivering at the same institution but not included in the prospective OB-CMI study. The NLP model was then compared with the OB-CMI in the subset with a known OB-CMI score. Model discrimination between the 2 approaches was compared using the DeLong test. Sensitivity and positive predictive value for the identification of individuals at highest risk were prioritized as the characteristics of interest. Results: This study included 19 794 individuals; 4034 (20.4%) were included in the original prospective validation study of the OB-CMI (testing set), and the remaining 15 760 (79.6%) composed the training set. Mean (SD) age was 32.3 (5.2) years in the testing cohort and 32.2 (5.2) years in the training cohort. A total of 115 individuals in the testing cohort (2.9%) and 468 in the training cohort (3.0%) experienced SMM. The NLP model was built from a pruned vocabulary of 2783 unique words that occurred within the 15 760 admission notes from individuals in the training set. The area under the receiver operating characteristic curve of the NLP-based model for the prediction of SMM was 0.76 (95% CI, 0.72-0.81) and was comparable with that of the OB-CMI model (0.74; 95% CI, 0.70-0.79) in the testing set (P = .53). Sensitivity (NLP, 28.7%; OB-CMI, 24.4%) and positive predictive value (NLP, 19.4%; OB-CMI, 17.6%) were comparable between the NLP and OB-CMI high-risk designations for the prediction of SMM. Conclusions and relevance: In this study, the NLP method and a validated risk-stratification tool had a similar ability to identify patients at high risk of SMM. Future prospective research is needed to validate the NLP approach in clinical practice and determine whether it could augment or replace tools requiring manual user input.

Natural Language Processing for Cohort Discovery in a Discharge Prediction Model for the Neonatal ICU

Predicting Discharge Dates From the NICU Using Progress Note Data

Nursing notes-aware deep neural network for predicting mortality risk after hospital discharge

Developing a length of stay prediction model for newborns, achieving better accuracy with greater usability

Comparison of Natural Language Processing of Clinical Notes With a Validated Risk-Stratification Tool to Predict Severe Maternal Morbidity

Development and Validation of a Machine Learning Model to Aid Discharge Processes for Inpatient Surgical Care

Abstract 13965: Natural Language Processing of Hospitalization Discharge Summary to Predict 1-year Post-Discharge Mortality Among Patients With Acute Heart Failure

Improving Prediction of Survival for Extremely Premature Infants Born at 23 to 29 Weeks Gestational Age in the Neonatal Intensive Care Unit: Development and Evaluation of Machine Learning Models

LCD Benchmark: Long Clinical Document Benchmark on Mortality Prediction

Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation

LCD Benchmark: Long Clinical Document Benchmark on Mortality Prediction for Language Models

Drug Clearance in Neonates: A Combination of Population Pharmacokinetic Modelling and Machine Learning Approaches to Improve Individual Prediction

Classifying early infant feeding status from clinical notes using natural language processing and machine learning

Explainable Machine Learning for ICU Readmission Prediction

Natural language processing of clinical notes enables early inborn error of immunity risk ascertainment

Clinical large language model to predict loss to follow up for oncology patients discharged to skilled nursing facilities.

Development of a low-dimensional model to predict admissions from triage at a pediatric emergency department

Predicting whether patients in an acute medical unit are physiologically fit-for-discharge using machine learning: A proof-of-concept

In-hospital mortality, readmission, and prolonged length of stay risk prediction leveraging historical electronic patient records

Predicting Admissions From a Paediatric Emergency Department - Protocol for Developing and Validating a Low-Dimensional Machine Learning Prediction Model

Traditional Machine Learning, Deep Learning, and BERT (Large Language Model) Approaches for Predicting Hospitalizations From Nurse Triage Notes: Comparative Evaluation of Resource Management