GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Pengcheng Chen,Jin Ye,Guoan Wang,Yanjun Li,Zhongying Deng,Wei Li,Tianbin Li,Haodong Duan,Ziyan Huang,Yanzhou Su,Benyou Wang,Shaoting Zhang,Bin Fu,Jianfei Cai,Bohan Zhuang,Eric J Seibel,Junjun He,Yu Qiao

2024-10-01

Abstract:Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 53.96%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the evaluation of the application of large visual language models (LVLMs) in the medical field. Specifically:

1. **Comprehensive Medical Knowledge**: Collected 284 clinically relevant datasets of different modalities from around the world, covering 38 different medical imaging modalities to ensure the model can handle diverse medical data.

2. **Well-Structured Data Classification**: Designed a vocabulary tree structure containing 18 clinical question-answering tasks, 18 clinical departments, and 4 levels of perceptual granularity, allowing users to customize evaluation tasks based on specific needs.

3. **Multi-Perceptual Granularity**: Evaluated the model's performance at different levels of perceptual detail, such as image-level and region-level, which is particularly important for diagnosis, segmentation, and classification tasks.

Through these designs, the GMAI-MMBench benchmark can more comprehensively evaluate the effectiveness of LVLMs in real clinical scenarios, identify the shortcomings of existing models, and provide guidance for future improvements. Experimental results show that even the most advanced models, such as GPT-4, can only achieve an accuracy of 53.96% on this benchmark, indicating significant room for improvement.