Abstract:Objectives: This study aims to assess the performance of a multimodal artificial intelligence (AI) model capable of analyzing both images and textual data (GPT-4V), in interpreting radiological images. It focuses on a range of modalities, anatomical regions, and pathologies to explore the potential of zero-shot generative AI in enhancing diagnostic processes in radiology. Methods: We analyzed 230 anonymized emergency room diagnostic images, consecutively collected over 1 week, using GPT-4V. Modalities included ultrasound (US), computerized tomography (CT), and X-ray images. The interpretations provided by GPT-4V were then compared with those of senior radiologists. This comparison aimed to evaluate the accuracy of GPT-4V in recognizing the imaging modality, anatomical region, and pathology present in the images. Results: GPT-4V identified the imaging modality correctly in 100% of cases (221/221), the anatomical region in 87.1% (189/217), and the pathology in 35.2% (76/216). However, the model's performance varied significantly across different modalities, with anatomical region identification accuracy ranging from 60.9% (39/64) in US images to 97% (98/101) and 100% (52/52) in CT and X-ray images (p < 0.001). Similarly, pathology identification ranged from 9.1% (6/66) in US images to 36.4% (36/99) in CT and 66.7% (34/51) in X-ray images (p < 0.001). These variations indicate inconsistencies in GPT-4V's ability to interpret radiological images accurately. Conclusion: While the integration of AI in radiology, exemplified by multimodal GPT-4, offers promising avenues for diagnostic enhancement, the current capabilities of GPT-4V are not yet reliable for interpreting radiological images. This study underscores the necessity for ongoing development to achieve dependable performance in radiology diagnostics. Clinical relevance statement: Although GPT-4V shows promise in radiological image interpretation, its high diagnostic hallucination rate (> 40%) indicates it cannot be trusted for clinical use as a standalone tool. Improvements are necessary to enhance its reliability and ensure patient safety. Key points: GPT-4V's capability in analyzing images offers new clinical possibilities in radiology. GPT-4V excels in identifying imaging modalities but demonstrates inconsistent anatomy and pathology detection. Ongoing AI advancements are necessary to enhance diagnostic reliability in radiological applications.

Potential of GPT-4 for Detecting Errors in Radiology Reports: Implications for Reporting Accuracy

GPT-4V Cannot Generate Radiology Reports Yet

Exploring the Boundaries of GPT-4 in Radiology

Assessing GPT-4 multimodal performance in radiological image analysis

Enhancing radiology training with GPT-4: Pilot analysis of automated feedback in trainee preliminary reports

Evaluation of GPT-4 ability to identify and generate patient instructions for actionable incidental radiology findings

Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential

The virtual reference radiologist: comprehensive AI assistance for clinical image reading and interpretation

A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports

Generative Large Language Models for Detection of Speech Recognition Errors in Radiology Reports

Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential

Evaluation of GPT Large Language Model Performance on RSNA 2023 Case of the Day Questions

A Comprehensive Study of GPT-4V's Multimodal Capabilities in Medical Imaging

GPT-4 Vision: Multi-Modal Evolution of ChatGPT and Potential Role in Radiology

Cutting Edge to Cutting Time: Can ChatGPT Improve the Radiologist's Reporting?

Advancing radiology with GPT-4: Innovations in clinical applications, patient engagement, research, and learning

Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation

Evaluation of GPT-4 for chest X-ray impression generation: A reader study on performance and perception

Evaluating GPT-4o's Performance in the Official European Board of Radiology Exam: A Comprehensive Assessment

Large language models (LLMs) in radiology exams for medical students: Performance and consequences