Boosting LLM-Assisted Diagnosis: 10-Minute LLM Tutorial Elevates Radiology Residents' Performance in Brain MRI Interpretation

Su Hwan Kim,Severin Schramm,Jonas Wihl,Philipp Raffler,Marlene Tahedl,Julian Canisius,Ina Luiken,Lukas Endroes,Stefan Reischl,Alexander Marka,Robert Walter,Mathias Schillmaier,Claus Zimmer,Benedikt Wiestler,Dennis Martin Hedderich

DOI: https://doi.org/10.1101/2024.07.03.24309779

2024-07-05

Abstract:Purpose To evaluate the impact of a structured tutorial on the use of a large language model (LLM)-based search engine on radiology residents' performance in LLM-assisted brain MRI differential diagnosis. Materials & Methods In this retrospective study, nine radiology residents determined the three most likely differential diagnoses for three sets of ten brain MRI cases with a challenging yet definite diagnosis. Each set of cases was assessed 1) with the support of conventional internet search, 2) using an LLM-based search engine (© Perplexity AI) without prior training, or 3) with LLM assistance after a structured 10-minute tutorial on how to effectively use the tool for differential diagnosis. The tutorial content was based on the results of two studies on LLM-assisted radiological diagnosis and included a prompt template. Reader responses were rated using a binary and numeric scoring system. Reading times were tracked and confidence levels were recorded on a 5-point Likert scale. Binary and numeric scores were analyzed using chi-square tests and pairwise Mann-Whitney U tests each. Search engine logs were examined to quantify user interaction metrics, and to identify hallucinations and misinterpretations in LLM responses. Results Radiology residents achieved the highest accuracy when employing the LLM-based search engine following the tutorial, indicating the correct diagnosis among the top three differential diagnoses in 62.5% of cases (55/88). This was followed by the LLM-assisted workflow before the tutorial (44.8%; 39/87) and the conventional internet search workflow (32.2%; 28/87). The LLM tutorial led to significantly higher performance (binary scores: p = 0.042, numeric scores: p = 0.016) and confidence (p = 0.006) but resulted in no relevant differences in reading times. Hallucinations were found in 5.1% of LLM queries. Conclusion A structured 10-minute LLM tutorial increased performance and confidence levels in LLM-assisted brain MRI differential diagnosis among radiology residents.