Abstract:While Large Vision-Language Models (LVLMs) have exhibited remarkable capabilities across a wide range of tasks, they suffer from hallucination problems, where models generate plausible yet incorrect answers given the input image-query pair. This hallucination phenomenon is even more severe when querying the image in non-English languages, while existing methods for mitigating hallucinations in LVLMs only consider the English scenarios. In this paper, we make the first attempt to mitigate this important multilingual hallucination in LVLMs. With thorough experiment analysis, we found that multilingual hallucination in LVLMs is a systemic problem that could arise from deficiencies in multilingual capabilities or inadequate multimodal abilities. To this end, we propose a two-stage Multilingual Hallucination Removal (MHR) framework for LVLMs, aiming to improve resistance to hallucination for both high-resource and low-resource languages. Instead of relying on the intricate manual annotations of multilingual resources, we fully leverage the inherent capabilities of the LVLM and propose a novel cross-lingual alignment method, which generates multiple responses for each image-query input and then identifies the hallucination-aware pairs for each language. These data pairs are finally used for direct preference optimization to prompt the LVLMs to favor non-hallucinating responses. Experimental results show that our MHR achieves a substantial reduction in hallucination generation for LVLMs. Notably, on our extended multilingual POPE benchmark, our framework delivers an average increase of 19.0% in accuracy across 13 different languages. Our code and model weights are available at <a class="link-external link-https" href="https://github.com/ssmisya/MHR" rel="external noopener nofollow">this https URL</a>

Skip : A Simple Method to Reduce Hallucination in Large Vision-Language Models

Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models

Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models

Hallucination of Multimodal Large Language Models: A Survey

Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback

Reducing Hallucinations in Vision-Language Models via Latent Space Steering

Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding

Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence

Analyzing and Mitigating Object Hallucination in Large Vision-Language Models

A Survey of Hallucination in Large Visual Language Models

Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization

Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding

A Survey on Hallucination in Large Vision-Language Models

Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning

Evaluation and Analysis of Hallucination in Large Vision-Language Models

Mitigating Multilingual Hallucination in Large Vision-Language Models

A Unified Hallucination Mitigation Framework for Large Vision-Language Models