Ultrasound investigation prior to organ donation.

E. M. Lang,T. Lang,D. Henne-Bruns,B. Kremer

IF: 1.014

1990-04-01

Transplantation Proceedings

Abstract:

What problem does this paper attempt to address?

Identifying stroke-related quantified evidence from electronic health records in real-world studies

Lin Yang,Xiaoshuo Huang,Jiayang Wang,Xin Yang,Lingling Ding,Zixiao Li,Jiao Li

DOI: https://doi.org/10.1016/j.artmed.2023.102552

Abstract:Background: Stroke is one of the leading causes of death and disability worldwide. The National Institutes of Health Stroke Scale (NIHSS) scores in electronic health records (EHRs), which quantitatively describe patients' neurological deficits in evidence-based treatment, are crucial in stroke-related clinical investigations. However, the free-text format and lack of standardization inhibit their effective use. Automatically extracting the scale scores from the clinical free text so that its potential value in real-world studies is realized has become an important goal. Objective: This study aims to develop an automated method to extract scale scores from the free text of EHRs. Methods: We propose a two-step pipeline method to identify NIHSS items and numerical scores and validate its feasibility using a freely accessible critical care database: MIMIC-III (Medical Information Mart for Intensive Care III). First, we utilize MIMIC-III to create an annotated corpus. Then, we investigate possible machine learning methods for two subtasks, NIHSS item and score recognition and item-score relation extraction. In the evaluation, we conduct both task-specific and end-to-end evaluations and compare our method with the rule-based method using precision, recall and F1 scores as evaluation metrics. Results: We use all available discharge summaries of stroke cases in MIMIC-III. The annotated NIHSS corpus contains 312 cases, 2929 scale items, 2774 scores and 2733 relations. The results show that the best F1-score of our method was 0.9006, which was attained by combining BERT-BiLSTM-CRF and Random Forest, and it outperformed the rule-based method (F1-score = 0.8098). In the end-to-end task, our method could successfully recognize the item "1b level of consciousness questions", the score "1" and their relation "('1b level of consciousness questions', '1', 'has value')" from the sentence "1b level of consciousness questions: said name = 1", while the rule-based method could not. Conclusions: The two-step pipeline method we propose is an effective approach to identify NIHSS items, scores and their relations. With its help, clinical investigators can easily retrieve and access structured scale data, thereby supporting stroke-related real-world studies.
Prediction-Driven Decision Support for Patients With Mild Stroke: A Model Based on Machine Learning Algorithms

Xinping Lin,Shiteng Lin,XiaoLi Cui,Daizun Zou,FuPing Jiang,JunShan Zhou,NiHong Chen,Zhihong Zhao,Juan Zhang,Jianjun Zou

DOI: https://doi.org/10.3389/fneur.2021.761092

IF: 3.4

2021-12-23

Frontiers in Neurology

Abstract:Background and Purpose: Treatment for mild stroke remains an open question. We aim to develop a decision support tool based on machine learning (ML) algorithms, called DAMS (Disability After Mild Stroke), to identify mild stroke patients who would be at high risk of post-stroke disability (PSD) if they only received medical therapy and, more importantly, to aid neurologists in making individual clinical decisions in emergency contexts. Methods: Ischemic stroke patients were prospectively recorded in the National Advanced Stroke Center of Nanjing First Hospital (China) between July 2016 and September 2020. The exclusion criteria were patients who received thrombolytic therapy, age <18 years, lack of 3-month modified Rankin Scale (mRS), disabled before the index stroke, with an admission National Institute of Health stroke scale (NIHSS) > 5. The primary outcome was PSD, corresponding to 3-month mRS ≥ 2. We developed five ML models and assessed the area under curve (AUC) of receiver operating characteristic, calibration curve, and decision curve analysis. The optimal ML model was selected to be DAMS. In addition, SHapley Additive exPlanations (SHAP) approach was introduced to rank the feature importance. Finally, rapid-DAMS (R-DAMS) was constructed for a more urgent situation based on DAMS. Results: A total of 1,905 mild stroke patients were enrolled in this study, and patients with PSD accounted for 23.4% (447). There was no difference in AUCs between the five models (ranged from 0.691 to 0.823). Although there was similar discriminative performance between ML models, the support vector machine model exhibited higher net benefit and better calibration (Brier score, 0.159, calibration slope, 0.935, calibration intercept, 0.035). Therefore, this model was selected for DAMS. In addition, SHAP approach showed that the most crucial feature was NIHSS on admission. Finally, R-DAMS was constructed and there was similar discriminative performance between R-DAMS and DAMS, but the former performed worse on calibration. Conclusions: DAMS and R-DAMS, as prediction-driven decision support tools, were designed to aid clinical decision-making for mild stroke patients in emergency contexts. In addition, even within a narrow range of baseline scores, NIHSS on admission is the strongest feature that contributed to the prediction.

neurosciences,clinical neurology
Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model

Zhanzhong Gu,Xiangjian He,Ping Yu,Wenjing Jia,Xiguang Yang,Gang Peng,Penghui Hu,Shiyan Chen,Hongjie Chen,Yiguang Lin

DOI: https://doi.org/10.1016/j.artmed.2024.102822

IF: 7.011

2024-02-28

Artificial Intelligence in Medicine

Abstract:Background: Stroke is a prevalent disease with a significant global impact. Effective assessment of stroke severity is vital for an accurate diagnosis, appropriate treatment, and optimal clinical outcomes. The National Institutes of Health Stroke Scale (NIHSS) is a widely used scale for quantitatively assessing stroke severity. However, the current manual scoring of NIHSS is labor-intensive, time-consuming, and sometimes unreliable. Applying artificial intelligence (AI) techniques to automate the quantitative assessment of stroke on vast amounts of electronic health records (EHRs) has attracted much interest. Objective: This study aims to develop an automatic, quantitative stroke severity assessment framework through automating the entire NIHSS scoring process on Chinese clinical EHRs. Methods: Our approach consists of two major parts: Chinese clinical named entity recognition (CNER) with a domain-adaptive pre-trained large language model (LLM) and automated NIHSS scoring. To build a high-performing CNER model, we first construct a stroke-specific, densely annotated dataset "Chinese Stroke Clinical Records" (CSCR) from EHRs provided by our partner hospital, based on a stroke ontology that defines semantically related entities for stroke assessment. We then pre-train a Chinese clinical LLM coined "CliRoberta" through domain-adaptive transfer learning and construct a deep learning-based CNER model that can accurately extract entities directly from Chinese EHRs. Finally, an automated, end-to-end NIHSS scoring pipeline is proposed by mapping the extracted entities to relevant NIHSS items and values, to quantitatively assess the stroke severity. Results: Results obtained on a benchmark dataset CCKS2019 and our newly created CSCR dataset demonstrate the superior performance of our domain-adaptive pre-trained LLM and the CNER model, compared with the existing benchmark LLMs and CNER models. The high F1 score of 0.990 ensures the reliability of our model in accurately extracting the entities for the subsequent automatic NIHSS scoring. Subsequently, our automated, end-to-end NIHSS scoring approach achieved excellent inter-rater agreement (0.823) and intraclass consistency (0.986) with the ground truth and significantly reduced the processing time from minutes to a few seconds. Conclusion: Our proposed automatic and quantitative framework for assessing stroke severity demonstrates exceptional performance and reliability through directly scoring the NIHSS from diagnostic notes in Chinese clinical EHRs. Moreover, this study also contributes a new clinical dataset, a pre-trained clinical LLM, and an effective deep learning-based CNER model. The deployment of these advanced algorithms can improve the accuracy and efficiency of clinical assessment, and help improve the quality, affordability and productivity of healthcare services.

engineering, biomedical,computer science, artificial intelligence,medical informatics
Abstract MP15: Validation of Phenotyping Algorithms for Stroke from Electronic Health Records Using Natural Language Processing

Yiqing Zhao,Suyang Fu,Nicholas B. Larson,Paul A. Decker,Alanna M. Chamberlain,Veronique L. Roger,Hongfang Liu,Suzette J. Bielinski

DOI: https://doi.org/10.1161/circ.139.suppl_1.mp15

IF: 37.8

2019-01-01

Circulation

Abstract:Background: Accurate identification of stroke cases from electronic health record (EHR) is needed for efficient and valid clinical and epidemiology research. A number of studies have evaluated the validity of using ICD-9 and ICD-10 codes for stroke identification. Previous research results show that sensitivity and positive predictive value (PPV) for stroke identification is the lowest among all other cerebrovascular diseases, with notable differences observed by stroke pathological subtypes. While most prevalent cerebrovascular disease cases can be detected using 430-438/I60-I69 collectively, more accurate and comprehensive stroke phenotyping algorithm is needed to identify incident stroke cases from EHR. In our work, we compared case identification results using ICD-codes exclusively with ICD plus a natural language processing (NLP) algorithm. Methods: We developed the NLP part of our stroke algorithm using a list of expert provided stroke-related keywords, which covers transient ischemic attack (TIA), ischemic stroke, and hemorrhagic stroke. We validated the hybrid (ICD+NLP) algorithm and compared it with ICD-exclusive algorithm in a previously established atrial fibrillation cohort (n=5,062). Clinical notes and ICD codes of all patients after the incident AF event were reviewed by two nurse abstracters to confirm new stroke incidences. All past clinical notes and ICD codes of a subset of patients (n=402) were reviewed to confirm lifetime (prior and current) stroke cases. Manual abstraction results were considered the gold standard for evaluation of ICD only and ICD+NLP automatic extraction. Sensitivity and positive predictive values (PPV) of both algorithms were calculated. Results: Among 5,062 patients, 593 patients were confirmed to have suffered a stroke after atrial fibrillation while 4,469 patients were confirmed with no EHR evidence of stroke after AF. The ICD-exclusive algorithm had a sensitivity of 47.2% and a PPV of 47.8% for detecting new stroke incidences. The hybrid stroke algorithm achieved a sensitivity of 92.4% and a PPV of 60.6% for extraction of incident stroke. For extraction of lifetime stroke cases, the hybrid approach achieved a sensitivity of 93.9% and a PPV of 88.7%. Performance of new stroke incidence extraction is limited because past/current stroke mentions in clinical notes are difficult to distinguish by our NLP algorithm (may be a future direction). Conclusions: We developed and validated a stroke algorithm that performed well for identifying incident and lifetime stroke cases. The addition of NLP into the stroke algorithm improved the sensitivity and PPV of compared to an ICD-exclusive algorithm.
Abstract P259: Using Natural Language Processing and Machine Learning to Identify Incident Stroke from Electronic Health Records

Yiqing Zhao,Sunyang Fu,Suzette J. Bielinski,Paul A. Decker,Alanna M. Chamberlain,Véronique L. Roger,Hongfang Liu,Nicolas B Larson

DOI: https://doi.org/10.1161/circ.141.suppl_1.p259

IF: 37.8

2020-01-01

Circulation

Abstract:Background: The focus of most existing phenotyping algorithms based on electronic health record (EHR) data has been to accurately identify cases and non-cases of specific diseases. However, a more challenging task is to accurately identify disease incidence, as identifying the first occurrence of disease is more important for efficient and valid clinical and epidemiological research. Moreover, stroke is a challenging phenotype due to diagnosis difficulty and common miscoding. This task generally requires utilization of multiple types of EHR data (e.g., diagnoses and procedure codes, unstructured clinical notes) and a more robust algorithm integrating both natural language processing and machine learning. In this study, we developed and validated an EHR-based classifier to accurately identify stroke incidence among a cohort of atrial fibrillation (AF) patients Methods: We developed a stroke phenotyping algorithm using International Classification of Diseases, Ninth Revision (ICD-9) codes, Current Procedural Terminology (CPT) codes, and expert-provided keywords as model features. Structured data was extracted from Rochester Epidemiology Project (REP) database. Natural Language Processing (NLP) was used to extract and validate keyword occurrence in clinical notes. A window of ±30 days was considered when including/excluding keywords/codes into the input vector. Frequencies of keywords/codes were used as input feature sets for model training. Multiple competing models were trained using various combinations of feature sets and two machine learning algorithms: logistic regression and random forest. Training data were provided by two nurse abstractors and included validated stroke incidences from a previously established atrial fibrillation cohort. Precision, recall, and F-score of the algorithm were calculated to assess and compare model performances. Results: Among 4,914 patients with atrial fibrillation, 1,773 patients were screened. 3,141 patients had no stroke-related codes or keywords and were presumed to be free of stroke during follow-up. Among the screened patients, 740 had validated strokes and 1,033 did not have a stroke based on review of the EHR by trained nurse abstractors. The best performing stroke incidence phenotyping classifier utilized Keywords+ICD-9+CPT features using a random forest classifier, achieving a precision of 0.942, recall of 0.943, and F-score of 0.943. Conclusion: In conclusion, we developed and validated a stroke algorithm that performed well for identifying stroke incidence in an enriched population (AF cohort), which extends beyond the typical binary case/non-case stroke identification problem. Future work will involve testing the generalizability of this algorithm in a general population.
Development of a Natural Language Processing (NLP) model to automatically extract clinical data from electronic health records: results from an Italian comprehensive stroke center

Davide Badalotti,Akanksha Agrawal,Umberto Pensato,Giovanni Angelotti,Simona Marcheselli

DOI: https://doi.org/10.1016/j.ijmedinf.2024.105626

Abstract:Introduction: Data collection often relies on time-consuming manual inputs, with a vast amount of information embedded in unstructured texts such as patients' medical records and clinical notes. Our study aims to develop a pipeline that combines active learning (AL) and NLP techniques to enhance data extraction in an acute ischemic stroke cohort. Materials and methods: Consecutive acute ischemic stroke patients who received reperfusion therapies at IRCCS Humanitas Research Hospital were included. The Italian NLP Bidirectional Encoder Representations from Transformers (BERT) model was trained with AL to automatically extract clinical variables from electronic health text. Simulated active learning performances were evaluated on a set of labels representing patients' comorbidities, comparing Bayesian Uncertainty Sampling by Disagreement (BALD) and random text selection. Prognostic models predicting patients' functional outcomes using Gradient Boosting were trained on manually labelled and semi-automatically extracted data and their performance was compared. Results: The active learning process initially showed null performance until around 20% of texts were labelled, possibly due to root layers freezing in the BERT model, yet overall, active learning improves model learning efficiency across most comorbidities. Prognostic modelling showed no significant difference in performance between models trained on manually labelled versus semi-automatically extracted data, indicating effective prediction capabilities in both settings. Conclusions: We developed an efficient language model to automate the extraction of clinical data from Italian unstructured health texts in a cohort of ischemic stroke patients. In a preliminary analysis, we demonstrated its potential applicability for enhancing prediction model accuracy.
Natural Language Processing Enhances Prediction of Functional Outcome After Acute Ischemic Stroke

Sheng‐Feng Sung,Chih‐Hao Chen,Ru‐Chiou Pan,Ya‐Han Hu,Jiann‐Shing Jeng

DOI: https://doi.org/10.1161/jaha.121.023486

IF: 6.106

2021-12-21

Journal of the American Heart Association

Abstract:Background Conventional prognostic scores usually require predefined clinical variables to predict outcome. The advancement of natural language processing has made it feasible to derive meaning from unstructured data. We aimed to test whether using unstructured text in electronic health records can improve the prediction of functional outcome after acute ischemic stroke. Methods and Results Patients hospitalized for acute ischemic stroke were identified from 2 hospital stroke registries (3847 and 2668 patients, respectively). Prediction models developed using the first cohort were externally validated using the second cohort, and vice versa. Free text in the history of present illness and computed tomography reports was used to build machine learning models using natural language processing to predict poor functional outcome at 90 days poststroke. Four conventional prognostic models were used as baseline models. The area under the receiver operating characteristic curves of the model using history of present illness in the internal and external validation sets were 0.820 and 0.792, respectively, which were comparable to the National Institutes of Health Stroke Scale score (0.811 and 0.807). The model using computed tomography reports achieved area under the receiver operating characteristic curves of 0.758 and 0.658. Adding information from clinical text significantly improved the predictive performance of each baseline model in terms of area under the receiver operating characteristic curves, net reclassification improvement, and integrated discrimination improvement indices (all P <0.001). Swapping the study cohorts led to similar results. Conclusions By using natural language processing, unstructured text in electronic health records can provide an alternative tool for stroke prognostication, and even enhance the performance of existing prognostic scores.

cardiac & cardiovascular systems
Evaluating local open-source large language models for data extraction from unstructured reports on mechanical thrombectomy in patients with ischemic stroke

Aymen Meddeb,Philipe Ebert,Keno Kyrill Bressem,Dmitriy Desser,Andrea Dell'Orco,Georg Bohner,Justus F Kleine,Eberhard Siebert,Nils Grauhan,Marc A Brockmann,Ahmed Othman,Michael Scheel,Jawed Nawabi

DOI: https://doi.org/10.1136/jnis-2024-022078

IF: 4.8

2024-08-02

Journal of NeuroInterventional Surgery

Abstract:Background A study was undertaken to assess the effectiveness of open-source large language models (LLMs) in extracting clinical data from unstructured mechanical thrombectomy reports in patients with ischemic stroke caused by a vessel occlusion. Methods We deployed local open-source LLMs to extract data points from free-text procedural reports in patients who underwent mechanical thrombectomy between September 2020 and June 2023 in our institution. The external dataset was obtained from a second university hospital and comprised consecutive cases treated between September 2023 and March 2024. Ground truth labeling was facilitated by a human-in-the-loop (HITL) approach, with time metrics recorded for both automated and manual data extractions. We tested three models—Mixtral, Qwen, and BioMistral—assessing their performance on precision, recall, and F1 score across 15 clinical categories such as National Institute of Health Stroke Scale (NIHSS) scores, occluded vessels, and medication details. Results The study included 1000 consecutive reports from our primary institution and 50 reports from a secondary institution. Mixtral showed the highest precision, achieving 0.99 for first series time extraction and 0.69 for occluded vessel identification within the internal dataset. In the external dataset, precision ranged from 1.00 for NIHSS scores to 0.70 for occluded vessels. Qwen showed moderate precision with a high of 0.85 for NIHSS scores and a low of 0.28 for occluded vessels. BioMistral had the broadest range of precision, from 0.81 for first series times to 0.14 for medication details. The HITL approach yielded an average time savings of 65.6% per case, with variations from 45.95% to 79.56%. Conclusion This study highlights the potential of using LLMs for automated clinical data extraction from medical reports. Incorporating HITL annotations enhances precision and also ensures the reliability of the extracted data. This methodology presents a scalable privacy-preserving option that can significantly support clinical documentation and research endeavors.

surgery,neuroimaging
Automating Stroke Data Extraction From Free-Text Radiology Reports Using Natural Language Processing: Instrument Validation Study

Amy Y X Yu,Zhongyu A Liu,Chloe Pou-Prom,Kaitlyn Lopes,Moira K Kapral,Richard I Aviv,Muhammad Mamdani

DOI: https://doi.org/10.2196/24381

IF: 3.2

2021-05-04

JMIR Medical Informatics

Abstract:Background Diagnostic neurovascular imaging data are important in stroke research, but obtaining these data typically requires laborious manual chart reviews. Objective We aimed to determine the accuracy of a natural language processing (NLP) approach to extract information on the presence and location of vascular occlusions as well as other stroke-related attributes based on free-text reports. Methods From the full reports of 1320 consecutive computed tomography (CT), CT angiography, and CT perfusion scans of the head and neck performed at a tertiary stroke center between October 2017 and January 2019, we manually extracted data on the presence of proximal large vessel occlusion (primary outcome), as well as distal vessel occlusion, ischemia, hemorrhage, Alberta stroke program early CT score (ASPECTS), and collateral status (secondary outcomes). Reports were randomly split into training (n=921) and validation (n=399) sets, and attributes were extracted using rule-based NLP. We reported the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and the overall accuracy of the NLP approach relative to the manually extracted data. Results The overall prevalence of large vessel occlusion was 12.2%. In the training sample, the NLP approach identified this attribute with an overall accuracy of 97.3% (95.5% sensitivity, 98.1% specificity, 84.1% PPV, and 99.4% NPV). In the validation set, the overall accuracy was 95.2% (90.0% sensitivity, 97.4% specificity, 76.3% PPV, and 98.5% NPV). The accuracy of identifying distal or basilar occlusion as well as hemorrhage was also high, but there were limitations in identifying cerebral ischemia, ASPECTS, and collateral status. Conclusions NLP may improve the efficiency of large-scale imaging data collection for stroke surveillance and research.

medical informatics
Automating Ischemic Stroke Subtype Classification Using Machine Learning and Natural Language Processing

Ravi Garg,Elissa Oh,Andrew Naidech,Konrad Kording,Shyam Prabhakaran

DOI: https://doi.org/10.1016/j.jstrokecerebrovasdis.2019.02.004

IF: 2.677

2019-07-01

Journal of Stroke and Cerebrovascular Diseases

Abstract:OBJECTIVE: The manual adjudication of disease classification is time-consuming, error-prone, and limits scaling to large datasets. In ischemic stroke (IS), subtype classification is critical for management and outcome prediction. This study sought to use natural language processing of electronic health records (EHR) combined with machine learning methods to automate IS subtyping.METHODS: Among IS patients from an observational registry with TOAST subtyping adjudicated by board-certified vascular neurologists, we analyzed unstructured text-based EHR data including neurology progress notes and neuroradiology reports using natural language processing. We performed several feature selection methods to reduce the high dimensionality of the features and 5-fold cross validation to test generalizability of our methods and minimize overfitting. We used several machine learning methods and calculated the kappa values for agreement between each machine learning approach to manual adjudication. We then performed a blinded testing of the best algorithm against a held-out subset of 50 cases.RESULTS: Compared to manual classification, the best machine-based classification achieved a kappa of .25 using radiology reports alone, .57 using progress notes alone, and .57 using combined data. Kappa values varied by subtype being highest for cardioembolic (.64) and lowest for cryptogenic cases (.47). In the held-out test subset, machine-based classification agreed with rater classification in 40 of 50 cases (kappa .72).CONCLUSIONS: Automated machine learning approaches using textual data from the EHR shows agreement with manual TOAST classification. The automated pipeline, if externally validated, could enable large-scale stroke epidemiology research.

neurosciences,peripheral vascular disease
Assessment of Natural Language Processing Methods for Ascertaining the Expanded Disability Status Scale Score From the Electronic Health Records of Patients With Multiple Sclerosis: Algorithm Development and Validation Study

Zhen Yang,Chloé Pou-Prom,Ashley Jones,Michaelia Banning,David Dai,Muhammad Mamdani,Jiwon Oh,Tony Antoniou

DOI: https://doi.org/10.2196/25157

IF: 3.2

2022-01-12

JMIR Medical Informatics

Abstract:Background The Expanded Disability Status Scale (EDSS) score is a widely used measure to monitor disability progression in people with multiple sclerosis (MS). However, extracting and deriving the EDSS score from unstructured electronic health records can be time-consuming. Objective We aimed to compare rule-based and deep learning natural language processing algorithms for detecting and predicting the total EDSS score and EDSS functional system subscores from the electronic health records of patients with MS. Methods We studied 17,452 electronic health records of 4906 MS patients followed at one of Canada’s largest MS clinics between June 2015 and July 2019. We randomly divided the records into training (80%) and test (20%) data sets, and compared the performance characteristics of 3 natural language processing models. First, we applied a rule-based approach, extracting the EDSS score from sentences containing the keyword “EDSS.” Next, we trained a convolutional neural network (CNN) model to predict the 19 half-step increments of the EDSS score. Finally, we used a combined rule-based–CNN model. For each approach, we determined the accuracy, precision, recall, and F-score compared with the reference standard, which was manually labeled EDSS scores in the clinic database. Results Overall, the combined keyword-CNN model demonstrated the best performance, with accuracy, precision, recall, and an F-score of 0.90, 0.83, 0.83, and 0.83 respectively. Respective figures for the rule-based and CNN models individually were 0.57, 0.91, 0.65, and 0.70, and 0.86, 0.70, 0.70, and 0.70. Because of missing data, the model performance for EDSS subscores was lower than that for the total EDSS score. Performance improved when considering notes with known values of the EDSS subscores. Conclusions A combined keyword-CNN natural language processing model can extract and accurately predict EDSS scores from patient records. This approach can be automated for efficient information extraction in clinical and research settings.

medical informatics
Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

Zhongheng Zhang,Hongying Ni

DOI: https://doi.org/10.1016/j.jointm.2024.09.002

2024-01-01

Journal of Intensive Medicine

Abstract:The integration of large language models (LLMs) in clinical medicine, particularly in critical care, has introduced transformative capabilities for analyzing and managing complex medical information. This technical note explores the application of LLMs, such as generative pretrained transformer 4 (GPT-4) and Qwen-Chat, in interpreting electronic healthcare records to assist with rapid patient condition assessments, predict sepsis, and automate the generation of discharge summaries. The note emphasizes the significance of LLMs in processing unstructured data from electronic health records (EHRs), extracting meaningful insights, and supporting personalized medicine through nuanced understanding of patient histories. Despite the technical complexity of deploying LLMs in clinical settings, this document provides a comprehensive guide to facilitate the effective integration of LLMs into clinical workflows, focusing on the use of DashScope's application programming interface (API) services for judgment on patient prognosis and organ support recommendations based on natural language in EHRs. By illustrating practical steps and best practices, this work aims to lower the technical barriers for clinicians and researchers, enabling broader adoption of LLMs in clinical research and practice to enhance patient care and outcomes.
A Machine Learning Approach to Support Urgent Stroke Triage Using Administrative Data and Social Determinants of Health at Hospital Presentation: Retrospective Study

Min Chen,Xuan Tan,Rema Padman

DOI: https://doi.org/10.2196/36477

2023-01-30

Abstract:Background: The key to effective stroke management is timely diagnosis and triage. Machine learning (ML) methods developed to assist in detecting stroke have focused on interpreting detailed clinical data such as clinical notes and diagnostic imaging results. However, such information may not be readily available when patients are initially triaged, particularly in rural and underserved communities. Objective: This study aimed to develop an ML stroke prediction algorithm based on data widely available at the time of patients' hospital presentations and assess the added value of social determinants of health (SDoH) in stroke prediction. Methods: We conducted a retrospective study of the emergency department and hospitalization records from 2012 to 2014 from all the acute care hospitals in the state of Florida, merged with the SDoH data from the American Community Survey. A case-control design was adopted to construct stroke and stroke mimic cohorts. We compared the algorithm performance and feature importance measures of the ML models (ie, gradient boosting machine and random forest) with those of the logistic regression model based on 3 sets of predictors. To provide insights into the prediction and ultimately assist care providers in decision-making, we used TreeSHAP for tree-based ML models to explain the stroke prediction. Results: Our analysis included 143,203 hospital visits of unique patients, and it was confirmed based on the principal diagnosis at discharge that 73% (n=104,662) of these patients had a stroke. The approach proposed in this study has high sensitivity and is particularly effective at reducing the misdiagnosis of dangerous stroke chameleons (false-negative rate <4%). ML classifiers consistently outperformed the benchmark logistic regression in all 3 input combinations. We found significant consistency across the models in the features that explain their performance. The most important features are age, the number of chronic conditions on admission, and primary payer (eg, Medicare or private insurance). Although both the individual- and community-level SDoH features helped improve the predictive performance of the models, the inclusion of the individual-level SDoH features led to a much larger improvement (area under the receiver operating characteristic curve increased from 0.694 to 0.823) than the inclusion of the community-level SDoH features (area under the receiver operating characteristic curve increased from 0.823 to 0.829). Conclusions: Using data widely available at the time of patients' hospital presentations, we developed a stroke prediction model with high sensitivity and reasonable specificity. The prediction algorithm uses variables that are routinely collected by providers and payers and might be useful in underresourced hospitals with limited availability of sensitive diagnostic tools or incomplete data-gathering capabilities.
Machine Learning-Based Prediction of Stroke in Emergency Departments

Vida Abedi,Debdipto Misra,Durgesh Chaudhary,Venkatesh Avula,Clemens M. Schirmer,Jiang Li,Ramin Zand

DOI: https://doi.org/10.1177/17562864241239108

2024-01-01

Therapeutic Advances in Neurological Disorders

Abstract:Background: Stroke misdiagnosis, associated with poor outcomes, is estimated to occur in 9% of all stroke patients. Objectives: We hypothesized that machine learning (ML) could assist in the diagnosis of ischemic stroke in emergency departments (EDs). Design: The study was conducted and reported according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guidelines. We performed model development and prospective temporal validation, using data from pre- and post-COVID periods; we also performed a case study on a small cohort of previously misdiagnosed stroke patients. Methods: We used structured and unstructured electronic health records (EHRs) of 56,452 patient encounters from 13 hospitals in Pennsylvania, from September 2003 to January 2021. ML pipelines, including natural language processing, were created using pre-event clinical data and provider notes in the EDs. Results: Using pre-event information, our model’s area under the receiver operating characteristics curve (AUROC) ranged from 0.88 to 0.92 with a similar range accuracy (0.87–0.90). Using provider notes, we identified five models that reached a balanced performance in terms of AUROC, sensitivity, and specificity. Model AUROC ranged from 0.93 to 0.99. Model sensitivity and specificity reached 0.90 and 0.99, respectively. Four of the top five performing models were based on the post-COVID provider notes; however, no performance difference between models tested on pre- and post-COVID was observed. Conclusion: This study leveraged pre-event and at-encounter level EHR for stroke prediction. The results indicate that available clinical information can be used for building EHR-based stroke prediction models and ED stroke alert systems.

clinical neurology
[Principles of surgical treatment of diabetic neuropathic osteoarthropathy].

S. Kessler,T. Kalteis,A. Botzlar

Der Internist

Abstract:
Natural Language Processing of Radiology Reports to Detect Complications of Ischemic Stroke

Miller, Matthew I.,Orfanoudaki, Agni,Cronin, Michael,Saglam, Hanife,So Yeon Kim, Ivy,Balogun, Oluwafemi,Tzalidi, Maria,Vasilopoulos, Kyriakos,Fanaropoulou, Georgia,Fanaropoulou, Nina M.,Kalin, Jack,Hutch, Meghan,Prescott, Brenton R.,Brush, Benjamin,Benjamin, Emelia J.,Shin, Min,Mian, Asim,Greer, David M.,Smirnakis, Stelios M.,Ong, Charlene J.

DOI: https://doi.org/10.1007/s12028-022-01513-3

2022-05-09

Neurocritical Care

Abstract:Abstraction of critical data from unstructured radiologic reports using natural language processing (NLP) is a powerful tool to automate the detection of important clinical features and enhance research efforts. We present a set of NLP approaches to identify critical findings in patients with acute ischemic stroke from radiology reports of computed tomography (CT) and magnetic resonance imaging (MRI). We trained machine learning classifiers to identify categorical outcomes of edema, midline shift (MLS), hemorrhagic transformation, and parenchymal hematoma, as well as rule-based systems (RBS) to identify intraventricular hemorrhage (IVH) and continuous MLS measurements within CT/MRI reports. Using a derivation cohort of 2289 reports from 550 individuals with acute middle cerebral artery territory ischemic strokes, we externally validated our models on reports from a separate institution as well as from patients with ischemic strokes in any vascular territory. In all data sets, a deep neural network with pretrained biomedical word embeddings (BioClinicalBERT) achieved the highest discrimination performance for binary prediction of edema (area under precision recall curve [AUPRC] > 0.94), MLS (AUPRC > 0.98), hemorrhagic conversion (AUPRC > 0.89), and parenchymal hematoma (AUPRC > 0.76). BioClinicalBERT outperformed lasso regression (p < 0.001) for all outcomes except parenchymal hematoma (p = 0.755). Tailored RBS for IVH and continuous MLS outperformed BioClinicalBERT (p < 0.001) and linear regression, respectively (p < 0.001). Our study demonstrates robust performance and external validity of a core NLP tool kit for identifying both categorical and continuous outcomes of ischemic stroke from unstructured radiographic text data. Medically tailored NLP methods have multiple important big data applications, including scalable electronic phenotyping, augmentation of clinical risk prediction models, and facilitation of automatic alert systems in the hospital setting.

clinical neurology,critical care medicine
Automated information extraction from free‐text medical documents for stroke key performance indicators: a pilot study

Stephen Bacchi,Sam Gluck,Simon Koblar,Jim Jannes,Timothy Kleinig

DOI: https://doi.org/10.1111/imj.15678

2022-02-01

Internal Medicine Journal

Abstract:Automated information extraction might be able to assist with the collection of stroke key performance indicators (KPI). The feasibility of using natural language processing for classification-based KPI and datetime field extraction was assessed. Using free-text discharge summaries, random forest models achieved high levels of performance in classification tasks (area under the receiver operator curve 0.95-1.00). The datetime field extraction method was successful in 29 of 43 (67.4%) cases. Further studies are indicated.

medicine, general & internal
Classification of neurologic outcomes from medical notes using natural language processing

Marta B Fernandes,Navid Valizadeh,Haitham S Alabsi,Syed A Quadri,Ryan A Tesh,Abigail A Bucklin,Haoqi Sun,Aayushee Jain,Laura N Brenner,Elissa Ye,Wendong Ge,Sarah I Collens,Stacie Lin,Sudeshna Das,Gregory K Robbins,Sahar F Zafar,Shibani S Mukerji,M Brandon Westover

DOI: https://doi.org/10.1016/j.eswa.2022.119171

2023-03-15

Abstract:Neurologic disability level at hospital discharge is an important outcome in many clinical research studies. Outside of clinical trials, neurologic outcomes must typically be extracted by labor intensive manual review of clinical notes in the electronic health record (EHR). To overcome this challenge, we set out to develop a natural language processing (NLP) approach that automatically reads clinical notes to determine neurologic outcomes, to make it possible to conduct larger scale neurologic outcomes studies. We obtained 7314 notes from 3632 patients hospitalized at two large Boston hospitals between January 2012 and June 2020, including discharge summaries (3485), occupational therapy (1472) and physical therapy (2357) notes. Fourteen clinical experts reviewed notes to assign scores on the Glasgow Outcome Scale (GOS) with 4 classes, namely 'good recovery', 'moderate disability', 'severe disability', and 'death' and on the Modified Rankin Scale (mRS), with 7 classes, namely 'no symptoms', 'no significant disability', 'slight disability', 'moderate disability', 'moderately severe disability', 'severe disability', and 'death'. For 428 patients' notes, 2 experts scored the cases generating interrater reliability estimates for GOS and mRS. After preprocessing and extracting features from the notes, we trained a multiclass logistic regression model using LASSO regularization and 5-fold cross validation for hyperparameter tuning. The model performed well on the test set, achieving a micro average area under the receiver operating characteristic and F-score of 0.94 (95% CI 0.93-0.95) and 0.77 (0.75-0.80) for GOS, and 0.90 (0.89-0.91) and 0.59 (0.57-0.62) for mRS, respectively. Our work demonstrates that an NLP algorithm can accurately assign neurologic outcomes based on free text clinical notes. This algorithm increases the scale of research on neurological outcomes that is possible with EHR data.
Mining Clinical Notes for Physical Rehabilitation Exercise Information: Natural Language Processing Algorithm Development and Validation Study

Sonish Sivarajkumar,Fengyi Gao,Parker E. Denny,Bayan M. Aldhahwani,Shyam Visweswaran,Allyn Bove,Yanshan Wang

2024-03-16

Abstract:Post-stroke patient rehabilitation requires precise, personalized treatment plans. Natural Language Processing (NLP) offers potential to extract valuable exercise information from clinical notes, aiding in the development of more effective rehabilitation strategies. Objective: This study aims to develop and evaluate a variety of NLP algorithms to extract and categorize physical rehabilitation exercise information from the clinical notes of post-stroke patients treated at the University of Pittsburgh Medical Center. A cohort of 13,605 patients diagnosed with stroke was identified, and their clinical notes containing rehabilitation therapy notes were retrieved. A comprehensive clinical ontology was created to represent various aspects of physical rehabilitation exercises. State-of-the-art NLP algorithms were then developed and compared, including rule-based, machine learning-based algorithms, and large language model (LLM)-based algorithms (ChatGPT). Analysis was conducted on a dataset comprising 23,724 notes with detailed demographic and clinical characteristics. The rule-based NLP algorithm demonstrated superior performance in most areas, particularly in detecting the 'Right Side' location with an F1 score of 0.975, outperforming Gradient Boosting by 0.063. Gradient Boosting excelled in 'Lower Extremity' location detection (F1 score: 0.978), surpassing rule-based NLP by 0.023. It also showed notable performance in 'Passive Range of Motion' with an F1 score of 0.970, a 0.032 improvement over rule-based NLP. The rule-based algorithm efficiently handled 'Duration', 'Sets', and 'Reps' with F1 scores up to 0.65. LLM-based NLP, particularly ChatGPT with few-shot prompts, achieved high recall but generally lower precision and F1 scores. However, it notably excelled in 'Backward Plane' motion detection, achieving an F1 score of 0.846, surpassing the rule-based algorithm's 0.720.

Computation and Language,Artificial Intelligence
Large language models can effectively extract stroke and reperfusion audit data from medical free-text discharge summaries

Rudy Goh,Benjamin Cook,Brandon Stretton,Andrew Ec Booth,Shrirajh Satheakeerthy,Sarah Howson,Joshua Kovoor,Aashray Gupta,Sheryn Tan,W Taylor Kimberly,Andrew Moey,Wilson Vallat,John Maddison,Jarrod Marks,Samuel Gluck,Toby Gilbert,Jim Jannes,Timothy Kleinig,Stephen Bacchi

DOI: https://doi.org/10.1016/j.jocn.2024.110847

Abstract:Introduction: Audits are an integral part of effective modern healthcare. The collection of data for audits can be resource intensive. Large language models (LLM) may be able to assist. This pilot study aimed to assess the feasibility of using a LLM to extract stroke audit data from free-text medical documentation. Method: Discharge summaries from a one-month retrospective cohort of stroke admissions at a tertiary hospital were collected. A locally-deployed LLM, LLaMA3, was then used to extract a variety of routine stroke audit data from free-text discharge summaries. These data were compared to the previously collected human audit data in the statewide registry. Manual case note review was undertaken in cases of discordance. Results: Overall, there was a total of 144 data points that were extracted (9 data points for each of the 16 patients). The LLM was correct in 135/144 (93.8%) of individual datapoints. This performance included binary categorical, multiple-option categorical, datetime, and free-text extraction fields. Conclusions: LLM may be able to assist with the efficient collection of stroke audit data. Such approaches may be pursued in other specialties. Future studies should seek to examine the most effective way to deploy such approaches in conjunction with human auditors and researchers.

Ultrasound investigation prior to organ donation.

Identifying stroke-related quantified evidence from electronic health records in real-world studies

Prediction-Driven Decision Support for Patients With Mild Stroke: A Model Based on Machine Learning Algorithms

Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model

Abstract MP15: Validation of Phenotyping Algorithms for Stroke from Electronic Health Records Using Natural Language Processing

Abstract P259: Using Natural Language Processing and Machine Learning to Identify Incident Stroke from Electronic Health Records

Development of a Natural Language Processing (NLP) model to automatically extract clinical data from electronic health records: results from an Italian comprehensive stroke center

Natural Language Processing Enhances Prediction of Functional Outcome After Acute Ischemic Stroke

Evaluating local open-source large language models for data extraction from unstructured reports on mechanical thrombectomy in patients with ischemic stroke

Automating Stroke Data Extraction From Free-Text Radiology Reports Using Natural Language Processing: Instrument Validation Study

Automating Ischemic Stroke Subtype Classification Using Machine Learning and Natural Language Processing

Assessment of Natural Language Processing Methods for Ascertaining the Expanded Disability Status Scale Score From the Electronic Health Records of Patients With Multiple Sclerosis: Algorithm Development and Validation Study

Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

A Machine Learning Approach to Support Urgent Stroke Triage Using Administrative Data and Social Determinants of Health at Hospital Presentation: Retrospective Study

Machine Learning-Based Prediction of Stroke in Emergency Departments

[Principles of surgical treatment of diabetic neuropathic osteoarthropathy].

Natural Language Processing of Radiology Reports to Detect Complications of Ischemic Stroke

Automated information extraction from free‐text medical documents for stroke key performance indicators: a pilot study

Classification of neurologic outcomes from medical notes using natural language processing

Mining Clinical Notes for Physical Rehabilitation Exercise Information: Natural Language Processing Algorithm Development and Validation Study

Large language models can effectively extract stroke and reperfusion audit data from medical free-text discharge summaries