Evaluating General Vision-Language Models for Clinical Medicine

Yixing Jiang,Jesutofunmi A. Omiye,Cyril Zakka,Michael Moor,Haiwen Gui,Shayan Alipour,Seyed Shahabeddin Mousavi,Jonathan H. Chen,Pranav Rajpurkar,Roxana Daneshjou

DOI: https://doi.org/10.1101/2024.04.12.24305744

2024-04-18

Abstract:Recently emerging large multimodal models (LMMs) utilize various types of data modalities, including text and visual inputs to generate outputs. The incorporation of LMMs into clinical medicine presents unique challenges, including accuracy, reliability, and clinical relevance. Here, we explore clinical applications of GPT-4V, an LMM that has been proposed for use in medicine, in gastroenterology, radiology, dermatology, and United States Medical Licensing Examination (USMLE) test questions. We used standardized robust datasets with thousands of endoscopy images, chest x-ray, and skin lesions to benchmark GPT-4V’s ability to predict diagnoses. To assess bias, we also explored GPT-4V’s ability to determine Fitzpatrick skin tones with dermatology images. We found that GPT-4V is limited in performance across all four domains, resulting in decreased performance compared to previously published baseline models. The macro-average precision, recall, and F1-score for gastroenterology were 11.2%, 9.1% and 6.8% respectively. For radiology, the best performing task of identifying cardiomegaly had precision, recall, and F1-score of 28%, 94%, and 43% respectively. In dermatology, GPT-4V had an overall top-1 and top-3 diagnostic accuracy of 6.2% and 21% respectively. There was a significant accuracy drop when predicting images of darker skin tones ( ). GPT-4V accurately identified Fitzpatrick skin tones for 56.5% of images. For the multiple-choice-styled USMLE image-based test questions, GPT-4V had an accuracy of 59%. Our findings demonstrate that the current version of GPT-4V is limited in its diagnostic abilities across multiple image-based medical specialties. Future work should be done to explore LMM’s sensitivity to prompting as well as hybrid models that can combine LMM’s capabilities with other robust models.

Health Informatics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate the application capabilities of large - scale multimodal models (LMMs), especially GPT - 4V, in clinical medicine. Specifically, the paper focuses on the performance of GPT - 4V in the following medical fields: 1. **Gastroenterology**: Evaluate GPT - 4V's ability to identify normal findings, anatomical landmarks, and pathological findings in endoscopic images. 2. **Radiology**: Evaluate GPT - 4V's ability to diagnose diseases such as atelectasis, cardiac enlargement, consolidation, edema, and pleural effusion in chest X - rays. 3. **Dermatology**: Evaluate GPT - 4V's ability to diagnose in skin lesion images, especially its robustness to different skin colors (Fitzpatrick skin types). 4. **United States Medical Licensing Examination (USMLE)**: Evaluate GPT - 4V's ability to answer multiple - choice questions based on images. ### Main problems - **Accuracy**: What is the diagnostic accuracy of GPT - 4V in each of the above medical fields? - **Reliability**: Are GPT - 4V's diagnostic results reliable? Can it provide explanations that meet clinical expectations? - **Clinical relevance**: Does GPT - 4V's performance meet the standards for clinical applications? Can it assist doctors in making decisions? - **Bias**: Does GPT - 4V have bias when processing images of different skin colors? ### Specific evaluation indicators - **Gastroenterology**: - Macro Average Precision - Macro Average Recall - Macro Average F1 Score - Micro Average Precision, Recall, F1 Score - Matthews Correlation Coefficient (MCC) - **Radiology**: - Sensitivity - Specificity - Precision - F1 Score - **Dermatology**: - Top - 1 and Top - 3 diagnostic accuracy - Diagnostic performance for different Fitzpatrick skin types - Predictive ability for malignant diseases - **USMLE**: - Accuracy ### Conclusion Through these evaluation indicators, the paper demonstrates the performance of GPT - 4V in multiple medical fields and points out its limitations in terms of accuracy, reliability, and clinical relevance. In particular, GPT - 4V has significant bias when processing images of different skin colors, especially performing poorly when identifying images of darker skin colors. These findings provide directions for future research, especially in improving the robustness of multimodal models and reducing bias.

Evaluating General Vision-Language Models for Clinical Medicine

Capability of GPT-4V(ision) in Japanese National Medical Licensing Examination

Comparative Analysis of GPT-4Vision, GPT-4 and Open Source LLMs in Clinical Diagnostic Accuracy: A Benchmark Against Human Expertise

Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models

Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis

Visual-Textual Integration in LLMs for Medical Diagnosis: A Quantitative Analysis

Performance of Multimodal GPT-4V on USMLE with Image: Potential for Imaging Diagnostic Support with Explanations

Diagnostic Accuracy of GPT Multimodal Analysis on USMLE Questions Including Text and Visuals

GPT-4V(ision) Unsuitable for Clinical Care and Education: A Clinician-Evaluated Assessment

Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis

Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine

Evaluating Large Language Models in Ophthalmology

Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V

Multimodal Large Language Models are Generalist Medical Image Interpreters

Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports

Large language models (LLMs) in radiology exams for medical students: Performance and consequences

Evaluation of large language models as a diagnostic aid for complex medical cases

Step into the era of large multimodal models: a pilot study on ChatGPT-4V(ision)'s ability to interpret radiological images

Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4

The Pulse of Artificial Intelligence in Cardiology: A Comprehensive Evaluation of State-of-the-art Large Language Models for Potential Use in Clinical Cardiology

Multimodal LLMs for Retinal Disease Diagnosis via OCT: Few-Shot vs Single-Shot Learning