Abstract:While large vision-language models (LVLMs) have shown impressive capabilities in generating plausible responses correlated with input visual contents, they still suffer from hallucinations, where the generated text inaccurately reflects visual contents. To address this, recent approaches apply contrastive decoding to calibrate the model's response via contrasting output distributions with original and visually distorted samples, demonstrating promising hallucination mitigation in a training-free manner. However, the potential of changing information in visual inputs is not well-explored, so a deeper investigation into the behaviors of visual contrastive decoding is of great interest. In this paper, we first explore various methods for contrastive decoding to change visual contents, including image downsampling and editing. Downsampling images reduces the detailed textual information while editing yields new contents in images, providing new aspects as visual contrastive samples. To further study benefits by using different contrastive samples, we analyze probability-level metrics, including entropy and distribution distance. Interestingly, the effect of these samples in mitigating hallucinations varies a lot across LVLMs and benchmarks. Based on our analysis, we propose a simple yet effective method to combine contrastive samples, offering a practical solution for applying contrastive decoding across various scenarios. Extensive experiments are conducted to validate the proposed fusion method among different benchmarks.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of "hallucinations" in large vision - language models (LVLMs) when generating text related to the input visual content. Specifically, LVLMs sometimes generate text that is semantically coherent but does not match the content of the input image, which may lead to the generation of misinformation. To alleviate this problem, the author explores different contrastive decoding (CD) methods and proposes a new method that combines multiple contrastive samples to improve the accuracy of LVLMs and reduce hallucinations. #### Specific problem description 1. **Hallucination phenomenon**: Although LVLMs perform well in generating text related to visual content, they are still prone to hallucinations, that is, the generated text is smooth and semantically coherent but does not match the input visual content. 2. **Limitations of existing methods**: - Most existing hallucination - alleviation methods require additional datasets and fine - tuning, which increases the burden of labor costs and computational resources and is not flexible enough in application in specific fields. - Contrastive decoding (CD) is an effective method without the need for additional datasets and fine - tuning, but its potential has not been fully explored. 3. **Research objectives**: - Explore the impact of different types of visual contrastive samples (such as image down - sampling, editing, etc.) on alleviating hallucinations. - Analyze the effects of different contrastive samples and their performance differences on different benchmarks and LVLMs. - Propose a method that fuses multiple contrastive samples to provide a more general and effective solution. #### Main contributions 1. **Explore multiple visual contrastive samples**: The author studies multiple visual contrastive samples, including image down - sampling and editing, and analyzes their impact on alleviating hallucinations. 2. **In - depth analysis of the effects of contrastive samples**: Through metrics at the probability distribution level (such as entropy and distribution distance), the author analyzes in detail the performance of different contrastive samples on various tasks and datasets, revealing the unique advantages of each sample. 3. **Propose a fusion method**: Based on the above analysis, the author proposes a simple and effective method to combine different types of contrastive samples, using entropy - weighted fusion to automatically determine the degree of influence of contrastive decoding, so as to better alleviate the hallucination problem in practical applications. Through these studies, the author provides a more general and practical solution for hallucination alleviation in LVLMs, which is applicable to different datasets and tasks.

Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models

Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization

Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models

Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)

VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding

Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models

Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding

IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding

Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding

HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding

CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models

Hallucination Augmented Contrastive Learning for Multimodal Large Language Model

CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs

V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization

MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation

Mitigating Hallucinations of Large Language Models in Medical Information Extraction via Contrastive Decoding

Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination

Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback