Hao Zhang,Neil Jethani,Simon Jones,Nicholas Genes,Vincent J Major,Ian S Jaffe,Anthony B Cardillo,Noah Heilenbach,Nadia Fazal Ali,Luke J Bonanni,Andrew J Clayburn,Zain Khera,Erica C Sadler,Jaideep Prasad,Jamie Schlacter,Kevin Liu,Benjamin Silva,Sophie Montgomery,Eric J Kim,Jacob Lester,Theodore M Hill,Alba Avoricani,Ethan Chervonski,James Davydov,William Small,Eesha Chakravartty,Himanshu Grover,John A Dodson,Abraham A Brody,Yindalon Aphinyanaphongs,Arjun Masurkar,Narges Razavian

Abstract:Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Use of a Large Language Model to Assess Clinical Acuity of Adults in the ED

Evaluating Large Language Model‐Assisted Emergency Triage: A Comparison of Acuity Assessments by GPT‐4 and Medical Experts

Diagnostic Accuracy of a Large Language Model in Pediatric Case Studies

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Bespoke Large Language Models for Digital Triage Assistance in Mental Health Care

Evaluating the Diagnostic Performance of Large Language Models on Complex Multimodal Medical Cases

Towards Accurate Differential Diagnosis with Large Language Models

The Capability of Large Language Models to Measure Psychiatric Functioning

Large Language Models as Agents in the Clinic

Evaluating the use of large language models to provide clinical recommendations in the Emergency Department

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room

Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial

Evaluating large language models as agents in the clinic

Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores

Enhancing Early Detection of Cognitive Decline in the Elderly: A Comparative Study Utilizing Large Language Models in Clinical Notes

Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

Can Large Language Models Provide Emergency Medical Help Where There Is No Ambulance? A Comparative Study on Large Language Model Understanding of Emergency Medical Scenarios in Resource-Constrained Settings

Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes

Pilot Study of Large Language Models as an Age-Appropriate Explanatory Tool for Chronic Pediatric Conditions