In-hospital cardiac arrest: characteristics and outcome after implementation of systematic practice-oriented training.

E. Ramberg,E. Wolsk,Jeanie M Elkjær,H. Bülow

DOI: https://doi.org/10.1016/j.ajem.2014.04.009

2014-07-01

Abstract:

What problem does this paper attempt to address?

Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models

Mark Iscoe,Vimig Socrates,Aidan Gilson,Ling Chi,Huan Li,Thomas Huang,Thomas Kearns,Rachelle Perkins,Laura Khandjian,R. Andrew Taylor

DOI: https://doi.org/10.1111/acem.14883

2024-04-05

Academic Emergency Medicine

Abstract:Background Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large‐scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms. Objectives This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large‐scale research, public health surveillance, and EHR‐based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes. Methods The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task‐specific LLMs to perform the task of named entity recognition: a convolutional neural network‐based model (SpaCy) and a transformer‐based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level. Results A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note‐level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model. Conclusions The study demonstrated the utility of LLMs and transformer‐based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.

emergency medicine
Can natural language processing models extract and classify instances of interpersonal violence in mental healthcare electronic records: an applied evaluative study

Riley Botelle,Vishal Bhavsar,Giouliana Kadra-Scalzo,Aurelie Mascio,Marcus V Williams,Angus Roberts,Sumithra Velupillai,Robert Stewart

DOI: https://doi.org/10.1136/bmjopen-2021-052911

IF: 3.006

2022-02-16

BMJ Open

Abstract:Objective: This paper evaluates the application of a natural language processing (NLP) model for extracting clinical text referring to interpersonal violence using electronic health records (EHRs) from a large mental healthcare provider. Design: A multidisciplinary team iteratively developed guidelines for annotating clinical text referring to violence. Keywords were used to generate a dataset which was annotated (ie, classified as affirmed, negated or irrelevant) for: presence of violence, patient status (ie, as perpetrator, witness and/or victim of violence) and violence type (domestic, physical and/or sexual). An NLP approach using a pretrained transformer model, BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) was fine-tuned on the annotated dataset and evaluated using 10-fold cross-validation. Setting: We used the Clinical Records Interactive Search (CRIS) database, comprising over 500 000 de-identified EHRs of patients within the South London and Maudsley NHS Foundation Trust, a specialist mental healthcare provider serving an urban catchment area. Participants: Searches of CRIS were carried out based on 17 predefined keywords. Randomly selected text fragments were taken from the results for each keyword, amounting to 3771 text fragments from the records of 2832 patients. Outcome measures: We estimated precision, recall and F1 score for each NLP model. We examined sociodemographic and clinical variables in patients giving rise to the text data, and frequencies for each annotated violence characteristic. Results: Binary classification models were developed for six labels (violence presence, perpetrator, victim, domestic, physical and sexual). Among annotations affirmed for the presence of any violence, 78% (1724) referred to physical violence, 61% (1350) referred to patients as perpetrator and 33% (731) to domestic violence. NLP models' precision ranged from 89% (perpetrator) to 98% (sexual); recall ranged from 89% (victim, perpetrator) to 97% (sexual). Conclusions: State of the art NLP models can extract and classify clinical text on violence from EHRs at acceptable levels of scale, efficiency and accuracy.
Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

Zhongheng Zhang,Hongying Ni

DOI: https://doi.org/10.1016/j.jointm.2024.09.002

2024-01-01

Journal of Intensive Medicine

Abstract:The integration of large language models (LLMs) in clinical medicine, particularly in critical care, has introduced transformative capabilities for analyzing and managing complex medical information. This technical note explores the application of LLMs, such as generative pretrained transformer 4 (GPT-4) and Qwen-Chat, in interpreting electronic healthcare records to assist with rapid patient condition assessments, predict sepsis, and automate the generation of discharge summaries. The note emphasizes the significance of LLMs in processing unstructured data from electronic health records (EHRs), extracting meaningful insights, and supporting personalized medicine through nuanced understanding of patient histories. Despite the technical complexity of deploying LLMs in clinical settings, this document provides a comprehensive guide to facilitate the effective integration of LLMs into clinical workflows, focusing on the use of DashScope's application programming interface (API) services for judgment on patient prognosis and organ support recommendations based on natural language in EHRs. By illustrating practical steps and best practices, this work aims to lower the technical barriers for clinicians and researchers, enabling broader adoption of LLMs in clinical research and practice to enhance patient care and outcomes.
Using a natural language processing toolkit to classify electronic health records by psychiatric diagnosis

Alissa Hutto,Tarek M Zikry,Buck Bohac,Terra Rose,Jasmine Staebler,Janet Slay,C Ray Cheever,Michael R Kosorok,Rebekah P. Nash

DOI: https://doi.org/10.1177/14604582241296411

2024-10-29

Health Informatics Journal

Abstract:Health Informatics Journal, Volume 30, Issue 4, October-December 2024. Objective: We analyzed a natural language processing (NLP) toolkit's ability to classify unstructured EHR data by psychiatric diagnosis. Expertise can be a barrier to using NLP. We employed an NLP toolkit (CLARK) created to support studies led by investigators with a range of informatics knowledge. Methods: The EHR of 652 patients were manually reviewed to establish Depression and Substance Use Disorder (SUD) labeled datasets, which were split into training and evaluation datasets. We used CLARK to train depression and SUD classification models using training datasets; model performance was analyzed against evaluation datasets. Results: The depression model accurately classified 69% of records (sensitivity = 0.68, specificity = 0.70, F1 = 0.68). The SUD model accurately classified 84% of records (sensitivity = 0.56, specificity = 0.92, F1 = 0.57). Conclusion: The depression model performed a more balanced job, while the SUD model's high specificity was paired with a low sensitivity. NLP applications may be especially helpful when combined with a confidence threshold for manual review.

health care sciences & services,medical informatics
Classifying Characteristics of Opioid Use Disorder From Hospital Discharge Summaries Using Natural Language Processing

Melissa N Poulsen,Philip J Freda,Vanessa Troiani,Anahita Davoudi,Danielle L Mowery

DOI: https://doi.org/10.3389/fpubh.2022.850619

2022-05-09

Abstract:Background: Opioid use disorder (OUD) is underdiagnosed in health system settings, limiting research on OUD using electronic health records (EHRs). Medical encounter notes can enrich structured EHR data with documented signs and symptoms of OUD and social risks and behaviors. To capture this information at scale, natural language processing (NLP) tools must be developed and evaluated. We developed and applied an annotation schema to deeply characterize OUD and related clinical, behavioral, and environmental factors, and automated the annotation schema using machine learning and deep learning-based approaches. Methods: Using the MIMIC-III Critical Care Database, we queried hospital discharge summaries of patients with International Classification of Diseases (ICD-9) OUD diagnostic codes. We developed an annotation schema to characterize problematic opioid use, identify individuals with potential OUD, and provide psychosocial context. Two annotators reviewed discharge summaries from 100 patients. We randomly sampled patients with their associated annotated sentences and divided them into training (66 patients; 2,127 annotated sentences) and testing (29 patients; 1,149 annotated sentences) sets. We used the training set to generate features, employing three NLP algorithms/knowledge sources. We trained and tested prediction models for classification with a traditional machine learner (logistic regression) and deep learning approach (Autogluon based on ELECTRA's replaced token detection model). We applied a five-fold cross-validation approach to reduce bias in performance estimates. Results: The resulting annotation schema contained 32 classes. We achieved moderate inter-annotator agreement, with F1-scores across all classes increasing from 48 to 66%. Five classes had a sufficient number of annotations for automation; of these, we observed consistently high performance (F1-scores) across training and testing sets for drug screening (training: 91-96; testing: 91-94) and opioid type (training: 86-96; testing: 86-99). Performance dropped from training and to testing sets for other drug use (training: 52-65; testing: 40-48), pain management (training: 72-78; testing: 61-78) and psychiatric (training: 73-80; testing: 72). Autogluon achieved the highest performance. Conclusion: This pilot study demonstrated that rich information regarding problematic opioid use can be manually identified by annotators. However, more training samples and features would improve our ability to reliably identify less common classes from clinical text, including text from outpatient settings.
Development and Evaluation of Machine Learning Models for the Detection of Emergency Department Patients with Opioid Misuse from Clinical Notes

Usman Shahid,Natalie Parde,Dale L Smith,Grayson Dickinson,Joseph Bianco,Dillon Thorpe,Madhav Hota,Majid Afshar,Niranjan S Karnik,neeraj chhabra

DOI: https://doi.org/10.1101/2024.12.11.24318875

2024-12-12

Abstract:Objectives The accurate identification of Emergency Department (ED) encounters involving opioid misuse is critical for health services, research, and surveillance. We sought to develop natural language processing (NLP)-based models for the detection of ED encounters involving opioid misuse. Methods A sample of ED encounters enriched for opioid misuse was manually annotated and clinical notes extracted. We evaluated classic machine learning (ML) methods, fine-tuning of publicly available pretrained language models, and a previously developed convolutional neural network opioid classifier for use on hospitalized patients (SMART-AI). Performance was compared to ICD-10-CM codes. Both raw text and text transformed to the United Medical Language System were evaluated. Face validity was evaluated by term feature importance. Results There were 1123 encounters used for training, validation, and testing. Of the classic ML methods, XGBoost had the highest AU_PRC (0.936), accuracy (0.887), and F1 score (0.863) which outperformed ICD-10-CM codes [accuracy 0.870; F1 0.830]. Logistic regression, support vector machine, and XGBoost models had higher AU_PRC using transformed text, while decision trees performed better using raw text. Excluding XGBoost, fine-tuned pre-trained language models outperformed classic ML methods. The best performing model was the fine-tuned SMART-AI based model with domain adaptation [AU_PRC 0.948; accuracy 0.882; F1 0.851]. Explainability analyses showed the most predictive terms were "heroin", "opioids", "alcoholic intoxication, chronic", "cocaine", "opiates", and "suboxone". Conclusions NLP-based models outperform entry of ICD-10-CM diagnosis codes for the detection of ED encounters with opioid misuse. Fine tuning with domain adaptation for pre-trained language models resulted in improved performance.
Manganese Reduction by Microbes from Oxic Regions of the Lake Vanda (Antarctica) Water Column

B. Bratina,B. Stevenson,W. Green,T. Schmidt

DOI: https://doi.org/10.1128/AEM.64.10.3791-3797.1998

1998-10-01

Applied and Environmental Microbiology

Abstract:ABSTRACT Depth profiles of metals in Lake Vanda, a permanently ice-covered, stratified Antarctic lake, suggest the importance of particulate manganese oxides in the scavenging, transport, and release of metals. Since manganese oxides can be solubilized by manganese-reducing bacteria, microbially mediated manganese reduction was investigated in Lake Vanda. Microbes concentrated from oxic regions of the water column, encompassing a peak of soluble manganese [Mn(II)], reduced synthetic manganese oxides (MnO2) when incubated aerobically. Pure cultures of manganese-reducing bacteria were readily isolated from waters collected near the oxic Mn(II) peak. Based on phylogenetic analysis of the 16S rRNA gene sequence, most of the isolated manganese reducers belong to the genusCarnobacterium. Cultures of a phylogenetically representative strain of Carnobacterium reduced synthetic MnO2 in the presence of sodium azide, as was seen in field assays. Unlike anaerobes that utilize manganese oxides as terminal electron acceptors in respiration, isolates of the genusCarnobacterium reduced Mn(IV) via a diffusible compound under oxic conditions. The release of adsorbed trace metals accompanying the solubilization of manganese oxides may provide populations of Carnobacterium with a source of nutrients in this extremely oligotrophic environment.
Using Large Language Models to Annotate Complex Cases of Social Determinants of Health in Longitudinal Clinical Records

Alexandra Ralevski,Nadaa Taiyab,Michael Nossal,Lindsay Mico,Samantha N. Piekos,Jennifer Hadlock

DOI: https://doi.org/10.1101/2024.04.25.24306380

2024-04-27

Abstract:Social Determinants of Health (SDoH) are an important part of the exposome and are known to have a large impact on variation in health outcomes. In particular, housing stability is known to be intricately linked to a patient’s health status, and pregnant women experiencing housing instability (HI) are known to have worse health outcomes. Most SDoH information is stored in electronic health records (EHRs) as free text (unstructured) clinical notes, which traditionally required natural language processing (NLP) for automatic identification of relevant text or keywords. A patient’s housing status can be ambiguous or subjective, and can change from note to note or within the same note, making it difficult to use existing NLP solutions. New developments in NLP allow researchers to prompt LLMs to perform complex, subjective annotation tasks that require reasoning that previously could only be attempted by human annotators. For example, large language models (LLMs) such as GPT (Generative Pre-trained Transformer) enable researchers to analyze complex, unstructured data using simple prompts. We used a secure platform within a large healthcare system to compare the ability of GPT-3.5 and GPT-4 to identify instances of both current and past housing instability, as well as general housing status, from 25,217 notes from 795 pregnant women. Results from these LLMs were compared with results from manual annotation, a named entity recognition (NER) model, and regular expressions (RegEx). We developed a chain-of-thought prompt requiring evidence and justification for each note from the LLMs, to help maximize the chances of finding relevant text related to HI while minimizing hallucinations and false positives. Compared with GPT-3.5 and the NER model, GPT-4 had the highest performance and had a much higher recall (0.924) than human annotators (0.702) in identifying patients experiencing current or past housing instability, although precision was lower (0.850) compared with human annotators (0.971). In most cases, the evidence output by GPT-4 was similar or identical to that of human annotators, and there was no evidence of hallucinations in any of the outputs from GPT-4. Most cases where the annotators and GPT-4 differed were ambiguous or subjective, such as “living in an apartment with too many people”. We also looked at GPT-4 performance on de-identified versions of the same notes and found that precision improved slightly (0.936 original, 0.939 de-identified), while recall dropped (0.781 original, 0.704 de-identified). This work demonstrates that, while manual annotation is likely to yield slightly more accurate results overall, LLMs, when compared with manual annotation, provide a scalable, cost-effective solution with the advantage of greater recall. At the same time, further evaluation is needed to address the risk of missed cases and bias in the initial selection of housing-related notes. Additionally, while it was possible to reduce confabulation, signs of unusual justifications remained. Given these factors, together with changes in both LLMs and charting over time, this approach is not yet appropriate for use as a fully-automated process. However, these results demonstrate the potential for using LLMs for computer-assisted annotation with human review, reducing cost and increasing recall. More efficient methods for obtaining structured SDoH data can help accelerate inclusion of exposome variables in biomedical research, and support healthcare systems in identifying patients who could benefit from proactive outreach.

Public and Global Health
Large language models to identify social determinants of health in electronic health records

Marco Guevara,Shan Chen,Spencer Thomas,Tafadzwa L. Chaunzwa,Idalid Franco,Benjamin H. Kann,Shalini Moningi,Jack M. Qian,Madeleine Goldstein,Susan Harper,Hugo J. W. L. Aerts,Paul J. Catalano,Guergana K. Savova,Raymond H. Mak,Danielle S. Bitterman

DOI: https://doi.org/10.1038/s41746-023-00970-0

IF: 15.2

2024-01-12

npj Digital Medicine

Abstract:Social determinants of health (SDoH) play a critical role in patient outcomes, yet their documentation is often missing or incomplete in the structured data of electronic health records (EHRs). Large language models (LLMs) could enable high-throughput extraction of SDoH from the EHR to support research and clinical care. However, class imbalance and data limitations present challenges for this sparsely documented yet critical information. Here, we investigated the optimal methods for using LLMs to extract six SDoH categories from narrative text in the EHR: employment, housing, transportation, parental status, relationship, and social support. The best-performing models were fine-tuned Flan-T5 XL for any SDoH mentions (macro-F1 0.71), and Flan-T5 XXL for adverse SDoH mentions (macro-F1 0.70). Adding LLM-generated synthetic data to training varied across models and architecture, but improved the performance of smaller Flan-T5 models (delta F1 + 0.12 to +0.23). Our best-fine-tuned models outperformed zero- and few-shot performance of ChatGPT-family models in the zero- and few-shot setting, except GPT4 with 10-shot prompting for adverse SDoH. Fine-tuned models were less likely than ChatGPT to change their prediction when race/ethnicity and gender descriptors were added to the text, suggesting less algorithmic bias ( p < 0.05). Our models identified 93.8% of patients with adverse SDoH, while ICD-10 codes captured 2.0%. These results demonstrate the potential of LLMs in improving real-world evidence on SDoH and assisting in identifying patients who could benefit from resource support.

health care sciences & services,medical informatics
Accuracy of a Proprietary Large Language Model in Labeling Obstetric Incident Reports

Jeanene Johnson,Conner Brown,Grace Lee,Keith Morse

DOI: https://doi.org/10.1016/j.jcjq.2024.08.001

2024-08-06

Abstract:Background: Using the data collected through incident reporting systems is challenging, as it is a large volume of primarily qualitative information. Large language models (LLMs), such as ChatGPT, provide novel capabilities in text summarization and labeling that could support safety data trending and early identification of opportunities to prevent patient harm. This study assessed the capability of a proprietary LLM (GPT-3.5) to automatically label a cross-sectional sample of real-world obstetric incident reports. Methods: A sample of 370 incident reports submitted to inpatient obstetric units between December 2022 and May 2023 was extracted. Human-annotated labels were assigned by a clinician reviewer and considered gold standard. The LLM was prompted to label incident reports relying solely on its pretrained knowledge and information included in the prompt. Primary outcomes assessed were sensitivity, specificity, positive predictive value, and negative predictive value. A secondary outcome assessed the human-perceived quality of the model's justification for the label(s) applied. Results: The LLM demonstrated the ability to label incident reports with high sensitivity and specificity. The model applied a total of 79 labels compared to the reviewer's 49 labels. Overall sensitivity for the model was 85.7%, and specificity was 97.9%. Positive and negative predictive values were 53.2% and 99.6%, respectively. For 60.8% of labels, the reviewer approved of the model's justification for applying the label. Conclusion: The proprietary LLM demonstrated the ability to label obstetric incident reports with high sensitivity and specificity. LLMs offer the potential to enable more efficient use of data from incident reporting systems.
[Principles of surgical treatment of diabetic neuropathic osteoarthropathy].

S. Kessler,T. Kalteis,A. Botzlar

Der Internist

Abstract:
An Inversion of the Obstacle Problem and its Explicit Solution

H. Lewy

DOI: https://doi.org/10.1007/978-1-4612-2082-4_35

Abstract:
Scalable information extraction from free text electronic health records using large language models

Bowen Gu,Vivian Shao,Ziqian Liao,Valentina Carducci,Santiago Romero-Brufau,Jie Yang,Rishi Desai

DOI: https://doi.org/10.1101/2024.08.08.24311237

2024-08-10

Abstract:Background: A vast amount of potentially useful information such as description of patient symptoms, family, and social history is recorded as free-text notes in electronic health records (EHRs) but is difficult to reliably extract at scale, limiting their utility in research. This study aims to assess whether an "out of the box" implementation of open-source large language models (LLMs) without any fine-tuning can accurately extract social determinants of health (SDoH) data from free-text clinical notes. Methods: We conducted a cross-sectional study using EHR data from the Mass General Brigham (MGB) system, analyzing free-text notes for SDoH information. We selected a random sample of 200 patients and manually labeled nine SDoH aspects. Eight advanced open-source LLMs were evaluated against a baseline pattern-matching model. Two human reviewers provided the manual labels, achieving 93% inter-annotator agreement. LLM performance was assessed using accuracy metrics for overall, mentioned, and non-mentioned SDoH, and macro F1 scores. Results: LLMs outperformed the baseline pattern-matching approach, particularly for explicitly mentioned SDoH, achieving up to 40% higher Accuracy_mentioned. openchat_3.5 was the best performing model, surpassing the baseline in overall accuracy across all nine SDoH aspects. The refined pipeline with prompt engineering reduced hallucinations and improved accuracy. Conclusions: Open-source LLMs are effective and scalable tools for extracting SDoH from unstructured EHRs, surpassing traditional pattern-matching methods. Further refinement and domain-specific training could enhance their utility in clinical research and predictive analytics, improving healthcare outcomes and addressing health disparities

Health Informatics
Extracting social determinants of health from electronic health records using natural language processing: a systematic review

Braja G Patra,Mohit M Sharma,Veer Vekaria,Prakash Adekkanattu,Olga V Patterson,Benjamin Glicksberg,Lauren A Lepow,Euijung Ryu,Joanna M Biernacka,Al’ona Furmanchuk,Thomas J George,William Hogan,Yonghui Wu,Xi Yang,Jiang Bian,Myrna Weissman,Priya Wickramaratne,J John Mann,Mark Olfson,Thomas R Campion,Mark Weiner,Jyotishman Pathak

DOI: https://doi.org/10.1093/jamia/ocab170

2021-10-06

Journal of the American Medical Informatics Association

Abstract:Abstract Objective Social determinants of health (SDoH) are nonclinical dispositions that impact patient health risks and clinical outcomes. Leveraging SDoH in clinical decision-making can potentially improve diagnosis, treatment planning, and patient outcomes. Despite increased interest in capturing SDoH in electronic health records (EHRs), such information is typically locked in unstructured clinical notes. Natural language processing (NLP) is the key technology to extract SDoH information from clinical text and expand its utility in patient care and research. This article presents a systematic review of the state-of-the-art NLP approaches and tools that focus on identifying and extracting SDoH data from unstructured clinical text in EHRs. Materials and Methods A broad literature search was conducted in February 2021 using 3 scholarly databases (ACL Anthology, PubMed, and Scopus) following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. A total of 6402 publications were initially identified, and after applying the study inclusion criteria, 82 publications were selected for the final review. Results Smoking status (n = 27), substance use (n = 21), homelessness (n = 20), and alcohol use (n = 15) are the most frequently studied SDoH categories. Homelessness (n = 7) and other less-studied SDoH (eg, education, financial problems, social isolation and support, family problems) are mostly identified using rule-based approaches. In contrast, machine learning approaches are popular for identifying smoking status (n = 13), substance use (n = 9), and alcohol use (n = 9). Conclusion NLP offers significant potential to extract SDoH data from narrative clinical notes, which in turn can aid in the development of screening tools, risk prediction models, and clinical decision support systems.

information science & library science,computer science, information systems, interdisciplinary applications,health care sciences & services,medical informatics
Natural Language Processing and Machine Learning to Identify People Who Inject Drugs in Electronic Health Records

David Goodman-Meza,Amber Tang,Babak Aryanfar,Sergio Vazquez,Adam J Gordon,Michihiko Goto,Matthew Bidwell Goetz,Steven Shoptaw,Alex A T Bui

DOI: https://doi.org/10.1093/ofid/ofac471

2022-09-12

Abstract:Background: Improving the identification of people who inject drugs (PWID) in electronic medical records can improve clinical decision making, risk assessment and mitigation, and health service research. Identification of PWID currently consists of heterogeneous, nonspecific International Classification of Diseases (ICD) codes as proxies. Natural language processing (NLP) and machine learning (ML) methods may have better diagnostic metrics than nonspecific ICD codes for identifying PWID. Methods: We manually reviewed 1000 records of patients diagnosed with Staphylococcus aureus bacteremia admitted to Veterans Health Administration hospitals from 2003 through 2014. The manual review was the reference standard. We developed and trained NLP/ML algorithms with and without regular expression filters for negation (NegEx) and compared these with 11 proxy combinations of ICD codes to identify PWID. Data were split 70% for training and 30% for testing. We calculated diagnostic metrics and estimated 95% confidence intervals (CIs) by bootstrapping the hold-out test set. Best models were determined by best F-score, a summary of sensitivity and positive predictive value. Results: Random forest with and without NegEx were the best-performing NLP/ML algorithms in the training set. Random forest with NegEx outperformed all ICD-based algorithms. F-score for the best NLP/ML algorithm was 0.905 (95% CI, .786-.967) and 0.592 (95% CI, .550-.632) for the best ICD-based algorithm. The NLP/ML algorithm had a sensitivity of 92.6% and specificity of 95.4%. Conclusions: NLP/ML outperformed ICD-based coding algorithms at identifying PWID in electronic health records. NLP/ML models should be considered in identifying cohorts of PWID to improve clinical decision making, health services research, and administrative surveillance.
Cognitive-Behavioral Group Therapy versus Sertraline for Obsessive-Compulsive Disorder: Five-Year Follow-Up

C. Borges,E. Meyer,Y. Ferrão,F. Souza,M. B. Sousa,A. Cordioli

DOI: https://doi.org/10.1159/000322028

2011-05-04

Psychotherapy and Psychosomatics

Abstract:ated 5 years after the end of the initial randomized clinical trial. All subjects provided written informed consent that was approved by the institutional review board. The Yale-Brown Scale for Obsessive-Compulsive Symptoms (Y-BOCS) [5, 6] and the Clinical Global Impression scale – Severity (CGI-S, adapted version) [7] were used to assess the severity of OCD symptoms. The Beck Depression Inventory (BDI) [8, 9] and Beck Anxiety Inventory (BAI) [10, 11] were used to assess depressive and anxiety symptoms, respectively. Both patients treated with CBGT and those treated with sertraline showed a significant reduction in the severity of symptoms at the end of the treatments, and they maintained therapeutic gains 5 years after. We also verified that a significant group of patients who underwent CBGT (13; 62%) started taking antiobsessional medication (p ! 0.001); on the other hand, psychotherapy was given to the patients who had taken sertraline (8; 32%), but at a nonsignificant level (p = 0.084). A total of 10 patients (22%), 2 (9.5%) from the CBGT group and 8 (32%) from the sertraline group, had CBT in the 5-year follow-up period ( 2 = 3.06; p = 0.084). Almost 62% of all patients continued using or started using medication, and 41.5% started or continued undergoing CBT in the 5-year follow-up period. There was a significant increase in remissive patients (partial or full remission) in the sertraline group (p = 0.046); the same did not occur in the CBGT group (p = 0.083). Table 1 shows patients’ scores on the Y-BOCS, CGI, BAI and BDI, as well as the nonimprovement, partial remission, full remission and relapse at the time of the follow-up study. The Y-BOCS total score median rate of variation between the initial treatment trial and the 5-year follow-up in the CBGT group ( fig. 1 ) was a 42.9% increase, while in the sertraline group there was a reduction by 17% (U Wilcoxon-Mann-Whitney = 167.5; p = 0.036). Cognitive-behavioral therapy (CBT) and serotonin reuptake inhibitors (SRI) are the first-choice treatments for obsessive-compulsive disorder (OCD) [1, 2] . Cognitive-behavioral group therapy (CBGT) has effectiveness similar to one-to-one CBT [3] . Speculation still persists whether the CBT outcomes, and especially CBGT, are sustained in the long term and if there are differences between the outcomes of CBT and antiobsessional medication. The aim of the present study was to investigate whether the reduction in severity of OCD symptoms obtained with 12-weekly sessions of CBGT or with sertraline (100 mg/day) during the same period, in a randomized clinical trial, would be sustained over a 5-year period, as well as to compare the differences between the 2 treatments in the long term. The randomized clinical trial in which patients were treated has been described in detail elsewhere [4] . Forty-six patients (92% of the initial sample) who were treated with CBGT (n = 21; 46%) or sertraline (n = 25; 54%) were evaluReceived: June 25, 2010 Accepted after revision: October 6, 2010 Published online: May 4, 2011
Large Language Models for Social Determinants of Health Information Extraction from Clinical Notes - A Generalizable Approach across Institutions

Vipina K Keloth,Salih Selek,Qingyu Chen,Christopher Gilman,Sunyang Fu,Yifang Dang,Xinghan Chen,Xinyue Hu,Yujia Zhou,Huan He,Jungwei W Fan,Karen Wang,Cynthia Brandt,Cui Tao,Hongfang Liu,Hua Xu

DOI: https://doi.org/10.1101/2024.05.21.24307726

2024-05-22

Abstract:The consistent and persuasive evidence illustrating the influence of social determinants on health has prompted a growing realization throughout the health care sector that enhancing health and health equity will likely depend, at least to some extent, on addressing detrimental social determinants. However, detailed social determinants of health (SDoH) information is often buried within clinical narrative text in electronic health records (EHRs), necessitating natural language processing (NLP) methods to automatically extract these details. Most current NLP efforts for SDoH extraction have been limited, investigating on limited types of SDoH elements, deriving data from a single institution, focusing on specific patient cohorts or note types, with reduced focus on generalizability. This study aims to address these issues by creating cross-institutional corpora spanning different note types and healthcare systems, and developing and evaluating the generalizability of classification models, including novel large language models (LLMs), for detecting SDoH factors from diverse types of notes from four institutions: Harris County Psychiatric Center, University of Texas Physician Practice, Beth Israel Deaconess Medical Center, and Mayo Clinic. Four corpora of deidentified clinical notes were annotated with 21 SDoH factors at two levels: level 1 with SDoH factor types only and level 2 with SDoH factors along with associated values. Three traditional classification algorithms (XGBoost, TextCNN, Sentence BERT) and an instruction tuned LLM-based approach (LLaMA) were developed to identify multiple SDoH factors. Substantial variation was noted in SDoH documentation practices and label distributions based on patient cohorts, note types, and hospitals. The LLM achieved top performance with micro-averaged F1 scores over 0.9 on level 1 annotated corpora and an F1 over 0.84 on level 2 annotated corpora. While models performed well when trained and tested on individual datasets, cross-dataset generalization highlighted remaining obstacles. To foster collaboration, access to partial annotated corpora and models trained by merging all annotated datasets will be made available on the PhysioNet repository.
Cohort Design and Natural Language Processing to Reduce Bias in Electronic Health Records Research: The Community Care Cohort Project

Shaan Khurshid,Christopher Reeder,Lia X. Harrington,Pulkit Singh,Gopal Sarma,Samuel F. Friedman,Paolo Di Achille,Nathaniel Diamant,Jonathan W. Cunningham,Ashby C. Turner,Emily S. Lau,Julian S. Haimovich,Mostafa A. Al-Alusi,Xin Wang,Marcus D.R. Klarqvist,Jeffrey M. Ashburner,Christian Diedrich,Mercedeh Ghadessi,Johanna Mielke,Hanna M. Eilken,Alice McElhinney,Andrea Derix,Steven J. Atlas,Patrick T. Ellinor,Anthony A. Philippakis,Christopher D. Anderson,Jennifer E. Ho,Puneet Batra,Steven A. Lubitz

DOI: https://doi.org/10.1101/2021.05.26.21257872

2021-05-30

Abstract:ABSTRACT Background Electronic health records (EHRs) promise to enable broad-ranging discovery with power exceeding that of conventional research cohort studies. However, research using EHR datasets may be subject to selection bias, which can be compounded by missing data, limiting the generalizability of derived insights. Methods Mass General Brigham (MGB) is a large New England-based healthcare network comprising seven tertiary care and community hospitals with associated outpatient practices. Within an MGB-based EHR warehouse of >3.5 million individuals with at least one ambulatory care visit, we approximated a community-based cohort study by selectively sampling individuals longitudinally attending primary care practices between 2001-2018 (n=520,868), which we named the Community Care Cohort Project (C3PO). We also utilized pre-trained deep natural language processing (NLP) models to recover vital signs (i.e., height, weight, and blood pressure) from unstructured notes in the EHR. We assessed the validity of C3PO by deploying established risk models including the Pooled Cohort Equations (PCE) and the Cohorts for Aging and Genomic Epidemiology Atrial Fibrillation (CHARGE-AF) score, and compared model performance in C3PO to that observed within typical EHR Convenience Samples which included all individuals from the same parent EHR with sufficient data to calculate each score but without a requirement for longitudinal primary care. All analyses were facilitated by the JEDI Extractive Data Infrastructure pipeline which we designed to efficiently aggregate EHR data within a unified framework conducive to regular updates. Results C3PO includes 520,868 individuals (mean age 48 years, 61% women, median follow-up 7.2 years, median primary care visits per individual 13). Estimated using reports, C3PO contains over 2.9 million electrocardiograms, 450,000 echocardiograms, 12,000 cardiac magnetic resonance images, and 75 million narrative notes. Using tabular data alone, 286,009 individuals (54.9%) had all vital signs available at baseline, which increased to 358,411 (68.8%) after NLP recovery (31% reduction in missingness). Among individuals with both NLP and tabular data available, NLP-extracted and tabular vital signs obtained on the same day were highly correlated (e.g., Pearson r range 0.95-0.99, p<0.01 for all). Both the PCE models (c-index range 0.724-0.770) and CHARGE-AF (c-index 0.782, 95% 0.777-0.787) demonstrated good discrimination. As compared to the Convenience Samples, AF and MI/stroke incidence rates in C3PO were lower and calibration error was smaller for both PCE (integrated calibration index range 0.012-0.030 vs. 0.028-0.046) and CHARGE-AF (0.028 vs. 0.036). Conclusions Intentional sampling of individuals receiving regular ambulatory care and use of NLP to recover missing data have the potential to reduce bias in EHR research and maximize generalizability of insights.
Classification of neurologic outcomes from medical notes using natural language processing

Marta B Fernandes,Navid Valizadeh,Haitham S Alabsi,Syed A Quadri,Ryan A Tesh,Abigail A Bucklin,Haoqi Sun,Aayushee Jain,Laura N Brenner,Elissa Ye,Wendong Ge,Sarah I Collens,Stacie Lin,Sudeshna Das,Gregory K Robbins,Sahar F Zafar,Shibani S Mukerji,M Brandon Westover

DOI: https://doi.org/10.1016/j.eswa.2022.119171

2023-03-15

Abstract:Neurologic disability level at hospital discharge is an important outcome in many clinical research studies. Outside of clinical trials, neurologic outcomes must typically be extracted by labor intensive manual review of clinical notes in the electronic health record (EHR). To overcome this challenge, we set out to develop a natural language processing (NLP) approach that automatically reads clinical notes to determine neurologic outcomes, to make it possible to conduct larger scale neurologic outcomes studies. We obtained 7314 notes from 3632 patients hospitalized at two large Boston hospitals between January 2012 and June 2020, including discharge summaries (3485), occupational therapy (1472) and physical therapy (2357) notes. Fourteen clinical experts reviewed notes to assign scores on the Glasgow Outcome Scale (GOS) with 4 classes, namely 'good recovery', 'moderate disability', 'severe disability', and 'death' and on the Modified Rankin Scale (mRS), with 7 classes, namely 'no symptoms', 'no significant disability', 'slight disability', 'moderate disability', 'moderately severe disability', 'severe disability', and 'death'. For 428 patients' notes, 2 experts scored the cases generating interrater reliability estimates for GOS and mRS. After preprocessing and extracting features from the notes, we trained a multiclass logistic regression model using LASSO regularization and 5-fold cross validation for hyperparameter tuning. The model performed well on the test set, achieving a micro average area under the receiver operating characteristic and F-score of 0.94 (95% CI 0.93-0.95) and 0.77 (0.75-0.80) for GOS, and 0.90 (0.89-0.91) and 0.59 (0.57-0.62) for mRS, respectively. Our work demonstrates that an NLP algorithm can accurately assign neurologic outcomes based on free text clinical notes. This algorithm increases the scale of research on neurological outcomes that is possible with EHR data.
Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language processing

Sifei Han,Robert F. Zhang,Lingyun Shi,Russell Richie,Haixia Liu,Andrew Tseng,Wei Quan,Neal Ryan,David Brent,Fuchiang R. Tsui

DOI: https://doi.org/10.1016/j.jbi.2021.103984

IF: 8

2022-03-01

Journal of Biomedical Informatics

Abstract:OBJECTIVE: Social determinants of health (SDOH) are non-medical factors that can profoundly impact patient health outcomes. However, SDOH are rarely available in structured electronic health record (EHR) data such as diagnosis codes, and more commonly found in unstructured narrative clinical notes. Hence, identifying social context from unstructured EHR data has become increasingly important. Yet, previous work on using natural language processing to automate extraction of SDOH from text (a) usually focuses on an ad hoc selection of SDOH, and (b) does not use the latest advances in deep learning. Our objective was to advance automatic extraction of SDOH from clinical text by (a) systematically creating a set of SDOH based on standard biomedical and psychiatric ontologies, and (b) training state-of-the-art deep neural networks to extract mentions of these SDOH from clinical notes.DESIGN: A retrospective cohort study.SETTING AND PARTICIPANTS: Data were extracted from the Medical Information Mart for Intensive Care (MIMIC-III) database. The corpus comprised 3,504 social related sentences from 2,670 clinical notes.METHODS: We developed a framework for automated classification of multiple SDOH categories. Our dataset comprised narrative clinical notes under the "Social Work" category in the MIMIC-III Clinical Database. Using standard terminologies, SNOMED-CT and DSM-IV, we systematically curated a set of 13 SDOH categories and created annotation guidelines for these. After manually annotating the 3,504 sentences, we developed and tested three deep neural network (DNN) architectures - convolutional neural network (CNN), long short-term memory (LSTM) network, and the Bidirectional Encoder Representations from Transformers (BERT) - for automated detection of eight SDOH categories. We also compared these DNNs to three baselines models: (1) cTAKES, as well as (2) L2-regularized logistic regression and (3) random forests on bags-of-words. Model evaluation metrics included micro- and macro- F1, and area under the receiver operating characteristic curve (AUC).RESULTS: All three DNN models accurately classified all SDOH categories (minimum micro-F1 = 0.632, minimum macro-AUC = 0.854). Compared to the CNN and LSTM, BERT performed best in most key metrics (micro-F1 = 0.690, macro-AUC = 0.907). The BERT model most effectively identified the "occupational" category (F1 = 0.774, AUC = 0.965) and least effectively identified the "non-SDOH" category (F = 0.491, AUC = 0.788). BERT outperformed cTAKES in distinguishing social vs non-social sentences (BERT F1 = 0.87 vs. cTAKES F1 = 0.06), and outperformed logistic regression (micro-F1 = 0.649, macro-AUC = 0.696) and random forest (micro-F1 = 0.502, macro-AUC = 0.523) trained on bag-of-words.CONCLUSIONS: Our study framework with DNN models demonstrated improved performance for efficiently identifying a systematic range of SDOH categories from clinical notes in the EHR. Improved identification of patient SDOH may further improve healthcare outcomes.

medical informatics,computer science, interdisciplinary applications

In-hospital cardiac arrest: characteristics and outcome after implementation of systematic practice-oriented training.

Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models

Can natural language processing models extract and classify instances of interpersonal violence in mental healthcare electronic records: an applied evaluative study

Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

Using a natural language processing toolkit to classify electronic health records by psychiatric diagnosis

Classifying Characteristics of Opioid Use Disorder From Hospital Discharge Summaries Using Natural Language Processing

Development and Evaluation of Machine Learning Models for the Detection of Emergency Department Patients with Opioid Misuse from Clinical Notes

Manganese Reduction by Microbes from Oxic Regions of the Lake Vanda (Antarctica) Water Column

Using Large Language Models to Annotate Complex Cases of Social Determinants of Health in Longitudinal Clinical Records

Large language models to identify social determinants of health in electronic health records

Accuracy of a Proprietary Large Language Model in Labeling Obstetric Incident Reports

[Principles of surgical treatment of diabetic neuropathic osteoarthropathy].

An Inversion of the Obstacle Problem and its Explicit Solution

Scalable information extraction from free text electronic health records using large language models

Extracting social determinants of health from electronic health records using natural language processing: a systematic review

Natural Language Processing and Machine Learning to Identify People Who Inject Drugs in Electronic Health Records

Cognitive-Behavioral Group Therapy versus Sertraline for Obsessive-Compulsive Disorder: Five-Year Follow-Up

Large Language Models for Social Determinants of Health Information Extraction from Clinical Notes - A Generalizable Approach across Institutions

Cohort Design and Natural Language Processing to Reduce Bias in Electronic Health Records Research: The Community Care Cohort Project

Classification of neurologic outcomes from medical notes using natural language processing

Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language processing