Abstract:Multi-modal Large Language Models (MLLMs) have demonstrated remarkable performance on various visual-language understanding and generation tasks. However, MLLMs occasionally generate content inconsistent with the given images, which is known as "hallucination". Prior works primarily center on evaluating hallucination using standard, unperturbed benchmarks, which overlook the prevalent occurrence of perturbed inputs in real-world scenarios-such as image cropping or blurring-that are critical for a comprehensive assessment of MLLMs' hallucination. In this paper, to bridge this gap, we propose Hallu-PI, the first benchmark designed to evaluate Hallucination in MLLMs within Perturbed Inputs. Specifically, Hallu-PI consists of seven perturbed scenarios, containing 1,260 perturbed images from 11 object types. Each image is accompanied by detailed annotations, which include fine-grained hallucination types, such as existence, attribute, and relation. We equip these annotations with a rich set of questions, making Hallu-PI suitable for both discriminative and generative tasks. Extensive experiments on 12 mainstream MLLMs, such as GPT-4V and Gemini-Pro Vision, demonstrate that these models exhibit significant hallucinations on Hallu-PI, which is not observed in unperturbed scenarios. Furthermore, our research reveals a severe bias in MLLMs' ability to handle different types of hallucinations. We also design two baselines specifically for perturbed scenarios, namely Perturbed-Reminder and Perturbed-ICL. We hope that our study will bring researchers' attention to the limitations of MLLMs when dealing with perturbed inputs, and spur further investigations to address this issue. Our code and datasets are publicly available at <a class="link-external link-https" href="https://github.com/NJUNLP/Hallu-PI" rel="external noopener nofollow">this https URL</a>.

AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation

An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation

Automated Multi-level Preference for MLLMs

LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models

Detecting and Evaluating Medical Hallucinations in Large Vision Language Models

Hallucination of Multimodal Large Language Models: A Survey

Evaluation and Analysis of Hallucination in Large Vision-Language Models

HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models

The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

Unified Hallucination Detection for Multimodal Large Language Models

Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models

Hallucination Augmented Contrastive Learning for Multimodal Large Language Model

HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation

Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs

ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models

Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models

Look Within, Why LLMs Hallucinate: A Causal Perspective