Abstract:Medical images and radiology reports are crucial for diagnosing medical conditions, highlighting the importance of quantitative analysis for clinical decision-making. However, the diversity and cross-source heterogeneity of these data challenge the generalizability of current data-mining methods. Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence (AGI) for computer vision, showcasing their potential in the biomedical domain. In this study, we evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets, including 5 medical imaging categories (dermatology, radiology, dentistry, ophthalmology, and endoscopy), and 3 radiology report datasets. The investigated tasks encompass disease classification, lesion segmentation, anatomical localization, disease diagnosis, report generation, and lesion detection. Our experimental results demonstrated that Gemini-series models excelled in report generation and lesion detection but faces challenges in disease classification and anatomical localization. Conversely, GPT-series models exhibited proficiency in lesion segmentation and anatomical localization but encountered difficulties in disease diagnosis and lesion detection. Additionally, both the Gemini series and GPT series contain models that have demonstrated commendable generation efficiency. While both models hold promise in reducing physician workload, alleviating pressure on limited healthcare resources, and fostering collaboration between clinical practitioners and artificial intelligence technologies, substantial enhancements and comprehensive validations remain imperative before clinical deployment.

Multimodal Large Language Models are Generalist Medical Image Interpreters

Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports

Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis

An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging

Evaluating General Vision-Language Models for Clinical Medicine

Comparison of Multi-Modal Large Language Models with Deep Learning Models for Medical Image Classification

Visual-Textual Integration in LLMs for Medical Diagnosis: A Quantitative Analysis

Capability of multimodal large language models to interpret pediatric radiological images

Multimodal Foundation Models Exploit Text to Make Medical Image Predictions

Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

Assessing large multimodal models for one-shot learning and interpretability in biomedical image classification

Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4

Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models

Evaluating Large Language Model (LLM) Performance on Established Breast Classification Systems

Multimodal Large Language Models for Bioimage Analysis

In-context learning enables multimodal large language models to classify cancer pathology images

Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundaries

On the Compositional Generalization of Multimodal LLMs for Medical Imaging

Large language models (LLMs) in radiology exams for medical students: Performance and consequences

From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice