Abstract:The remarkable generative capabilities of multimodal foundation models are currently being explored for a variety of applications. Generating radiological impressions is a challenging task that could significantly reduce the workload of radiologists. In our study we explored and analyzed the generative abilities of GPT-4 for Chest X-ray impression generation. To generate and evaluate impressions of chest X-rays based on different input modalities (image, text, text and image), a blinded radiological report was written for 25-cases of the publicly available NIH-dataset. GPT-4 was given image, finding section or both sequentially to generate an input dependent impression. In a blind randomized reading, 4-radiologists rated the impressions and were asked to classify the impression origin (Human, AI), providing justification for their decision. Lastly text model evaluation metrics and their correlation with the radiological score (summation of the 4 dimensions) was assessed. According to the radiological score, the human-written impression was rated highest, although not significantly different to text-based impressions. The automated evaluation metrics showed moderate to substantial correlations to the radiological score for the image impressions, however individual scores were highly divergent among inputs, indicating insufficient representation of radiological quality. Detection of AI-generated impressions varied by input and was 61% for text-based impressions. Impressions classified as AI-generated had significantly worse radiological scores even when written by a radiologist, indicating potential bias. Our study revealed significant discrepancies between a radiological assessment and common automatic evaluation metrics depending on the model input. The detection of AI-generated findings is subject to bias that highly rated impressions are perceived as human-written.

From Text to Image: Exploring GPT-4Vision's Potential in Advanced Radiological Analysis across Subspecialties

Evaluating GPT-4 with Vision on Detection of Radiological Findings on Chest Radiographs

Correction: Integrating Text and Image Analysis: Exploring GPT-4V's Capabilities in Advanced Radiological Applications Across Subspecialties

Evaluating GPT-V4 (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs

Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation

A Comprehensive Study of GPT-4V's Multimodal Capabilities in Medical Imaging

GPT-4 Vision: Multi-Modal Evolution of ChatGPT and Potential Role in Radiology

Toward Foundation Models in Radiology? Quantitative Assessment of GPT-4V's Multimodal and Multianatomic Region Capabilities

Exploring the Boundaries of GPT-4 in Radiology

Assessing GPT-4 Multimodal Performance in Radiological Image Analysis

Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis

Performance of GPT-4 with Vision on Text- and Image-based ACR Diagnostic Radiology In-Training Examination Questions

A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis

Comparative Analysis of GPT-4Vision, GPT-4 and Open Source LLMs in Clinical Diagnostic Accuracy: A Benchmark Against Human Expertise

Holistic Evaluation of GPT-4V for Biomedical Imaging

GPT-4 Vision on Medical Image Classification -- A Case Study on COVID-19 Dataset

Evaluation of GPT Large Language Model Performance on RSNA 2023 Case of the Day Questions

Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation

Keeping Up With ChatGPT

Evaluation of GPT-4 for chest X-ray impression generation: A reader study on performance and perception