Abstract:Recently emerging large multimodal models (LMMs) utilize various types of data modalities, including text and visual inputs to generate outputs. The incorporation of LMMs into clinical medicine presents unique challenges, including accuracy, reliability, and clinical relevance. Here, we explore clinical applications of GPT-4V, an LMM that has been proposed for use in medicine, in gastroenterology, radiology, dermatology, and United States Medical Licensing Examination (USMLE) test questions. We used standardized robust datasets with thousands of endoscopy images, chest x-ray, and skin lesions to benchmark GPT-4V’s ability to predict diagnoses. To assess bias, we also explored GPT-4V’s ability to determine Fitzpatrick skin tones with dermatology images. We found that GPT-4V is limited in performance across all four domains, resulting in decreased performance compared to previously published baseline models. The macro-average precision, recall, and F1-score for gastroenterology were 11.2%, 9.1% and 6.8% respectively. For radiology, the best performing task of identifying cardiomegaly had precision, recall, and F1-score of 28%, 94%, and 43% respectively. In dermatology, GPT-4V had an overall top-1 and top-3 diagnostic accuracy of 6.2% and 21% respectively. There was a significant accuracy drop when predicting images of darker skin tones ( ). GPT-4V accurately identified Fitzpatrick skin tones for 56.5% of images. For the multiple-choice-styled USMLE image-based test questions, GPT-4V had an accuracy of 59%. Our findings demonstrate that the current version of GPT-4V is limited in its diagnostic abilities across multiple image-based medical specialties. Future work should be done to explore LMM’s sensitivity to prompting as well as hybrid models that can combine LMM’s capabilities with other robust models.

Insight: A Multi-Modal Diagnostic Pipeline using LLMs for Ocular Surface Disease Diagnosis

Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis

Visual-Textual Integration in LLMs for Medical Diagnosis: A Quantitative Analysis

OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue

Multimodal LLMs for Retinal Disease Diagnosis via OCT: Few-Shot vs Single-Shot Learning

EyeGPT: Ophthalmic Assistant with Large Language Models

Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm

Understanding natural language: Potential application of large language models to ophthalmology

OphGLM: An ophthalmology large language-and-vision assistant

AI Hospital: Interactive Evaluation and Collaboration of LLMs As Intern Doctors for Clinical Diagnosis

LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models

Ophtha-LLaMA2: A Large Language Model for Ophthalmology

CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making

Development and evaluation of a large language model of ophthalmology in Chinese

EyeCLIP: A visual-language foundation model for multi-modal ophthalmic image analysis

The Role of Prompt Engineering for Multimodal LLM Glaucoma Diagnosis

MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications

Interpretable Differential Diagnosis with Dual-Inference Large Language Models

Evaluating General Vision-Language Models for Clinical Medicine

Towards Accurate Differential Diagnosis with Large Language Models

MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis