Abstract:Background: The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery. Objective: We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool. Methods: The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan. Results: The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR. Conclusions: The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence-driven CDS. Trial registration: ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480.

DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record

Deploying Predictive Models In A Healthcare Environment - An Open Source Approach

Clinical deployment environments: Five pillars of translational machine learning for health

Deploying clinical machine learning? Consider the following...

Scalable and accurate deep learning with electronic health records

Implementation of a Machine Learning Risk Prediction Model for Postpartum Depression in the Electronic Health Records

Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults

An operational guide to translational clinical machine learning in academic medical centers

Use of deep learning to develop continuous-risk models for adverse event prediction from electronic health records

EHR-ML: A generalisable pipeline for reproducible clinical outcomes using electronic health records

Clinical Deployment of Machine Learning Tools in Transplant Medicine: What Does the Future Hold?

Development & Deployment of a Real-time Healthcare Predictive Analytics Platform

Automating Construction of Machine Learning Models With Clinical Big Data: Proposal Rationale and Methods

Prospective silent deployment and evaluation of an intelligent machine learning model for prediction of emergency department visits during cancer treatment.

Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks

From compute to care: Lessons learned from deploying an early warning system into clinical practice

A scalable approach for developing clinical risk prediction applications in different hospitals

Discovery Viewer (DV): Web-Based Medical AI Model Development Platform and Deployment Hub

Monitoring Diagnostic Safety Risks in Emergency Departments: Protocol for a Machine Learning Study

Scalable Predictive Analysis in Critically Ill Patients Using a Visual Open Data Analysis Platform

Deployment of Deep Learning Model in Real World Clinical Setting: A Case Study in Obstetric Ultrasound