Abstract:Background: The diagnostic accuracy of differential diagnoses generated by artificial intelligence chatbots, including ChatGPT models, for complex clinical vignettes derived from general internal medicine (GIM) department case reports is unknown. Objective: This study aims to evaluate the accuracy of the differential diagnosis lists generated by both third-generation ChatGPT (ChatGPT-3.5) and fourth-generation ChatGPT (ChatGPT-4) by using case vignettes from case reports published by the Department of GIM of Dokkyo Medical University Hospital, Japan. Methods: We searched PubMed for case reports. Upon identification, physicians selected diagnostic cases, determined the final diagnosis, and displayed them into clinical vignettes. Physicians typed the determined text with the clinical vignettes in the ChatGPT-3.5 and ChatGPT-4 prompts to generate the top 10 differential diagnoses. The ChatGPT models were not specially trained or further reinforced for this task. Three GIM physicians from other medical institutions created differential diagnosis lists by reading the same clinical vignettes. We measured the rate of correct diagnosis within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and the top diagnosis. Results: In total, 52 case reports were analyzed. The rates of correct diagnosis by ChatGPT-4 within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and top diagnosis were 83% (43/52), 81% (42/52), and 60% (31/52), respectively. The rates of correct diagnosis by ChatGPT-3.5 within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and top diagnosis were 73% (38/52), 65% (34/52), and 42% (22/52), respectively. The rates of correct diagnosis by ChatGPT-4 were comparable to those by physicians within the top 10 (43/52, 83% vs 39/52, 75%, respectively; P =.47) and within the top 5 (42/52, 81% vs 35/52, 67%, respectively; P =.18) differential diagnosis lists and top diagnosis (31/52, 60% vs 26/52, 50%, respectively; P =.43) although the difference was not significant. The ChatGPT models' diagnostic accuracy did not significantly vary based on open access status or the publication date (before 2011 vs 2022). Conclusions: This study demonstrates the potential diagnostic accuracy of differential diagnosis lists generated using ChatGPT-3.5 and ChatGPT-4 for complex clinical vignettes from case reports published by the GIM department. The rate of correct diagnoses within the top 10 and top 5 differential diagnosis lists generated by ChatGPT-4 exceeds 80%. Although derived from a limited data set of case reports from a single department, our findings highlight the potential utility of ChatGPT-4 as a supplementary tool for physicians, particularly for those affiliated with the GIM department. Further investigations should explore the diagnostic accuracy of ChatGPT by using distinct case materials beyond its training data. Such efforts will provide a comprehensive insight into the role of artificial intelligence in enhancing clinical decision-making.

Decoding the Clavien-Dindo Classification

Decoding the Clavien-Dindo Classification: Artificial Intelligence (AI) as a Novel Tool to Grade Postoperative Complications

Analysis of ChatGPT in the Triage of Common Spinal Complaints

Evaluating Chatgpt As an Adjunct for Analyzing Challenging Case

P467 Towards AI-Augmented Clinical Decision Making: An Examination of ChatGPT's Utility in Acute Ulcerative Colitis Presentations

Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians

A Quantitative Assessment of ChatGPT as a Neurosurgical Triaging Tool

ChatGPT-Generated Differential Diagnosis Lists for Complex Case–Derived Clinical Vignettes: Diagnostic Accuracy Evaluation

Evaluating ChatGPT's moral competence in health care-related ethical problems

Analyzing the Performance of ChatGPT in Cardiology and Vascular Pathologies

AI integration in nephrology: evaluating ChatGPT for accurate ICD-10 documentation and coding

Investigating the clinical reasoning abilities of large language model GPT-4: an analysis of postoperative complications from renal surgeries

S2148 Assessment of ChatGPT-4's Efficacy in Providing Guideline-Directed Assistance to Residents and Fellows for Critical GI Conditions in Acute Settings

Unveiling the risks of ChatGPT in diagnostic surgical pathology

Assessment of ChatGPT's Compliance with ESC-Acute Coronary Syndrome Management Guidelines at 30-Day Intervals

ChatGPT Earns American Board Certification in Hand Surgery

Unveiling the risks of ChatGPT in diagnostic surgical pathologyChatGPT

Assessing ChatGPT 4.0's test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports

Human intelligence versus Chat-GPT: who performs better in correctly classifying patients in triage?

ChatGPT and retinal disease: a cross-sectional study on AI comprehension of clinical guidelines

Evaluating Performance of ChatGPT on MKSAP Cardiology Board Review Questions