Abstract:Background: Large language models (LLMs) have shown promising performance in various healthcare domains, but their effectiveness in identifying specific clinical conditions in real medical records is less explored. This study evaluates LLMs for detecting signs of cognitive decline in real electronic health record (EHR) clinical notes, comparing their error profiles with traditional models. The insights gained will inform strategies for performance enhancement. Methods: This study, conducted at Mass General Brigham in Boston, MA, analyzed clinical notes from the four years prior to a 2019 diagnosis of mild cognitive impairment in patients aged 50 and older. We used a randomly annotated sample of 4,949 note sections, filtered with keywords related to cognitive functions, for model development. For testing, a random annotated sample of 1,996 note sections without keyword filtering was utilized. We developed prompts for two LLMs, Llama 2 and GPT-4, on HIPAA-compliant cloud-computing platforms using multiple approaches (e.g., both hard and soft prompting and error analysis-based instructions) to select the optimal LLM-based method. Baseline models included a hierarchical attention-based neural network and XGBoost. Subsequently, we constructed an ensemble of the three models using a majority vote approach. Results: GPT-4 demonstrated superior accuracy and efficiency compared to Llama 2, but did not outperform traditional models. The ensemble model outperformed the individual models, achieving a precision of 90.3%, a recall of 94.2%, and an F1-score of 92.2%. Notably, the ensemble model showed a significant improvement in precision, increasing from a range of 70%-79% to above 90%, compared to the best-performing single model. Error analysis revealed that 63 samples were incorrectly predicted by at least one model; however, only 2 cases (3.2%) were mutual errors across all models, indicating diverse error profiles among them. Conclusions: LLMs and traditional machine learning models trained using local EHR data exhibited diverse error profiles. The ensemble of these models was found to be complementary, enhancing diagnostic performance. Future research should investigate integrating LLMs with smaller, localized models and incorporating medical data and domain knowledge to enhance performance on specific tasks.

WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Models

UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?

WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models

Improving Clinical Note Generation from Complex Doctor-Patient Conversation

Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notes

Adapting Open-Source Large Language Models for Cost-Effective, Expert-Level Clinical Note Generation with On-Policy Reinforcement Learning

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Retrieval-Augmented and Knowledge-Grounded Language Models for Faithful Clinical Medicine

Integrating UMLS Knowledge into Large Language Models for Medical Question Answering

Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundaries

Generalizable clinical note section identification with large language models

ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation

Enhancing Clinical Accuracy of Medical Chatbots with Large Language Models

[Clinical application of large language models : Does ChatGPT replace medical report formulation? An experience report]

Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding

PMC-LLaMA: Towards Building Open-source Language Models for Medicine

PMC-LLaMA: toward building open-source language models for medicine

Enhancing Early Detection of Cognitive Decline in the Elderly: A Comparative Study Utilizing Large Language Models in Clinical Notes

PULSAR at MEDIQA-Sum 2023: Large Language Models Augmented by Synthetic Dialogue Convert Patient Dialogues to Medical Records

PALLM: Evaluating and Enhancing PALLiative Care Conversations with Large Language Models