Abstract:Background: The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery. Objective: We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool. Methods: The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan. Results: The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR. Conclusions: The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence-driven CDS. Trial registration: ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480.

Automated‐detection of risky alcohol use prior to surgery using natural language processing

Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning study

A Computational Approach to Mortality Prediction of Alcohol Use Disorder Inpatients

Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults

Using natural language processing to identify opioid use disorder in electronic health record data

Automated machine learning‐based model predicts postoperative delirium using readily extractable perioperative collected electronic data

Use of natural language processing method to identify regional anesthesia from clinical notes

Identification of Alcohol Use Prior to Major Cancer Surgery: Timeline Follow Back Interview Compared to Four Other Markers

Identification of pancreatic cancer risk factors from clinical notes using natural language processing

Validation of a natural language processing algorithm using national reporting data to improve identification of anesthesia-related ADVerse evENTs: The "ADVENTURE" study

Deep Neural Network to Identify Patients with Alcohol Use Disorder

Individualized risk assessment of preoperative opioid use by interpretable neural network regression

Natural Language Processing and Machine Learning to Identify People Who Inject Drugs in Electronic Health Records

Using natural language processing in facilitating pre-hospital telephone triage of emergency calls

Early identification of patients with acute gastrointestinal bleeding using natural language processing and decision rules

Integrating machine learning predictions for perioperative risk management: Towards an empirical design of a flexible-standardized risk assessment tool

Postoperative delirium prediction using machine learning models and preoperative electronic health record data

Development and Evaluation of Machine Learning Models for the Detection of Emergency Department Patients with Opioid Misuse from Clinical Notes

A Combined Classification Model for Chinese Clinical Notes

History of alcohol abuse after major non-cardiac surgery and postoperative cognitive dysfunction

Development and validation of a novel AI framework using NLP with LLM integration for relevant clinical data extraction through automated chart review