MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context

Zishan Gu,Changchang Yin,Fenglin Liu,Ping Zhang

2024-07-03

Abstract:Large Vision Language Models (LVLMs) have recently achieved superior performance in various tasks on natural image and text data, which inspires a large amount of studies for LVLMs fine-tuning and training. Despite their advancements, there has been scant research on the robustness of these models against hallucination when fine-tuned on smaller datasets. In this study, we introduce a new benchmark dataset, the Medical Visual Hallucination Test (MedVH), to evaluate the hallucination of domain-specific LVLMs. MedVH comprises five tasks to evaluate hallucinations in LVLMs within the medical context, which includes tasks for comprehensive understanding of textual and visual input, as well as long textual response generation. Our extensive experiments with both general and medical LVLMs reveal that, although medical LVLMs demonstrate promising performance on standard medical tasks, they are particularly susceptible to hallucinations, often more so than the general models, raising significant concerns about the reliability of these domain-specific models. For medical LVLMs to be truly valuable in real-world applications, they must not only accurately integrate medical knowledge but also maintain robust reasoning abilities to prevent hallucination. Our work paves the way for future evaluations of these studies.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the issue of hallucinations in large-scale vision-language models (LVLMs) fine-tuned on small datasets when handling medical tasks. Specifically, although these models perform well on standard medical tasks, they are prone to generating incorrect information when faced with misleading inputs. To tackle this problem, the research team developed a new benchmark dataset called the Medical Visual Hallucination Test (MedVH), aimed at evaluating the resistance of these models to hallucinations in medical settings. MedVH includes 5 different types of tasks, covering various aspects from text-visual understanding to long text generation, to comprehensively assess the models' performance. Experimental results indicate that while domain-specific LVLMs perform well on certain standard tasks, they are still prone to errors when dealing with misleading inputs, posing challenges to their reliability in real-world medical applications. Through this study, the authors hope to promote the development of more reliable and trustworthy language models and facilitate their practical application in real medical scenarios.