Abstract:Background The use of large language models (LLM) has recently gained popularity in diverse areas, including answering questions posted by patients as well as medical professionals. Objective To evaluate the performance and limitations of LLMs in providing the correct diagnosis for a complex clinical case. Design Seventy-five consecutive clinical cases were selected from the Massachusetts General Hospital Case Records, and differential diagnoses were generated by OpenAI’s GPT3.5 and 4 models. Results The mean number of diagnoses provided by the Massachusetts General Hospital case discussants was 16.77, by GPT3.5 30 and by GPT4 15.45 ( p < 0.0001). GPT4 was more frequently able to list the correct diagnosis as first (22% versus 20% with GPT3.5, p = 0.86), provide the correct diagnosis among the top three generated diagnoses (42% versus 24%, p = 0.075). GPT4 was better at providing the correct diagnosis, when the different diagnoses were classified into groups according to the medical specialty and include the correct diagnosis at any point in the differential list (68% versus 48%, p = 0.0063). GPT4 provided a differential list that was more similar to the list provided by the case discussants than GPT3.5 (Jaccard Similarity Index 0.22 versus 0.12, p = 0.001). Inclusion of the correct diagnosis in the generated differential was correlated with PubMed articles matching the diagnosis (OR 1.40, 95% CI 1.25–1.56 for GPT3.5, OR 1.25, 95% CI 1.13–1.40 for GPT4), but not with disease incidence. Conclusions and relevance The GPT4 model was able to generate a differential diagnosis list with the correct diagnosis in approximately two thirds of cases, but the most likely diagnosis was often incorrect for both models. In its current state, this tool can at most be used as an aid to expand on potential diagnostic considerations for a case, and future LLMs should be trained which account for the discrepancy between disease incidence and availability in the literature.

Evaluating the Diagnostic Performance of Large Language Models on Complex Multimodal Medical Cases

Diagnostic Accuracy of a Large Language Model in Pediatric Case Studies

Towards Accurate Differential Diagnosis with Large Language Models

Evaluating large language models in medical applications: a survey

Combining Insights From Multiple Large Language Models Improves Diagnostic Accuracy

Large Language Models in Medical Term Classification and Unexpected Misalignment Between Response and Reasoning

Evaluation of large language models as a diagnostic aid for complex medical cases

Comparative Analysis of Multimodal Large Language Model Performance on Clinical Vignette Questions

MSDiagnosis: A Benchmark for Evaluating Large Language Models in Multi-Step Clinical Diagnosis

Large language models encode clinical knowledge

Interpretable Differential Diagnosis with Dual-Inference Large Language Models

Evaluating Large Language Models for Radiology Natural Language Processing

Large Language Models Illuminate a Progressive Pathway to Artificial Intelligent Healthcare Assistant

Testing the Limits of Language Models: A Conversational Framework for Medical AI Assessment

Multimodal Large Language Models are Generalist Medical Image Interpreters

Large Language Models are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales

Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator

Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

Large Language Models Perform Diagnostic Reasoning

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Large Language Models for Disease Diagnosis: A Scoping Review