Abstract:Background: The potential of large language models (LLM) such as GPT to support complex tasks such as differential diagnosis has been a subject of debate, with some ascribing near sentient abilities to the models and others claiming that LLMs merely perform "autocomplete on steroids". A recent study reported that the Generative Pretrained Transformer 4 (GPT-4) model performed well in complex differential diagnostic reasoning. The authors assessed the performance of GPT-4 in identifying the correct diagnosis in a series of case records from the New England Journal of Medicine. The authors constructed prompts based on the clinical presentation section of the case reports, and compared the results of GPT-4 to the actual diagnosis. GPT-4 returned the correct diagnosis as a part of its response in 64% of cases, with the correct diagnosis being at rank 1 in 39% of cases. However, such concise but comprehensive narratives of the clinical course are not typically available in electronic health records (EHRs). Further, if they were available, EHR records contain identifying information whose transmission is prohibited by Health Insurance Portability and Accountability Act (HIPAA) regulations. Methods: To assess the expected performance of GPT on comparable datasets that can be generated by text mining and by design cannot contain identifiable information, we parsed the texts of the case reports and extracted Human Phenotype Ontology (HPO) terms, from which prompts for GPT were constructed that contain largely the same clinical abnormalities but lack the surrounding narrative. Results: While the performance of GPT-4 on the original narrative-based text was good, with the final diagnosis being included in its differential in 29/75 cases (38.7%; rank 1 in 17.3% of cases; mean rank of 3.4), the performance of GPT-4 on the feature-based approach that includes the major clinical abnormalities without additional narrative texas substantially worse, with GPT-4 including the final diagnosis in its differential in 8/75 cases (10.7%; rank 1 in 4.0% of cases; mean rank of 3.9). Interpretation: We consider the feature-based queries to be a more appropriate test of the performance of GPT-4 in diagnostic tasks, since it is unlikely that the narrative approach can be used in actual clinical practice. Future research and algorithmic development is needed to determine the optimal approach to leveraging LLMs for clinical diagnosis.

CoD, Towards an Interpretable Medical Agent using Chain of Diagnosis

Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosis

Integrating Automated Knowledge Extraction with Large Language Models for Explainable Medical Decision-Making

Large Language Models Perform Diagnostic Reasoning

Deciphering Diagnoses: How Large Language Models Explanations Influence Clinical Decision Making

Interpretable Differential Diagnosis with Dual-Inference Large Language Models

DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models

Beyond Label Attention: Transparency in Language Models for Automated Medical Coding via Dictionary Learning

Large Language Models and Medical Knowledge Grounding for Diagnosis Prediction

DictLLM: Harnessing Key-Value Data Structures with Large Language Models for Enhanced Medical Diagnostics

CoD-MIL: Chain-of-Diagnosis Prompting Multiple Instance Learning for Whole Slide Image Classification.

Conversational Disease Diagnosis via External Planner-Controlled Large Language Models

ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models

Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias

Leveraging A Medical Knowledge Graph into Large Language Models for Diagnosis Prediction

MedCoT: Medical Chain of Thought Via Hierarchical Expert

Exploring LLM Multi-Agents for ICD Coding

A Pathology Image Diagnosis Network With Visual Interpretability And Structured Diagnostic Report

On the limitations of large language models in clinical diagnosis