Abstract:In the realms of computer vision and natural language processing, Large Vision-Language Models (LVLMs) have become indispensable tools, proficient in generating textual descriptions based on visual inputs. Despite their advancements, our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior rather than the input image. Our empirical experiments underscore the persistence of this bias, as LVLMs often provide confident answers even in the absence of relevant images or given incongruent visual input. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies. Firstly, for tasks such as classification or multi-choice question-answering (QA), we propose a ``calibration'' step through affine transformation to adjust the output distribution. This ``Post-Hoc debias'' approach ensures uniform scores for each answer when the image is absent, serving as an effective regularization technique to alleviate the influence of LLM priors. For more intricate open-ended generation tasks, we extend this method to ``Debias sampling'', drawing inspirations from contrastive decoding methods. Furthermore, our investigation sheds light on the instability of LVLMs across various decoding configurations. Through systematic exploration of different settings, we significantly enhance performance, surpassing reported results and raising concerns about the fairness of existing evaluations. Comprehensive experiments substantiate the effectiveness of our proposed strategies in mitigating biases. These strategies not only prove beneficial in minimizing hallucinations but also contribute to the generation of more helpful and precise illustrations.

Can We Debias Multimodal Large Language Models Via Model Editing?

Can We Edit Multimodal Large Language Models?

Potential and Challenges of Model Editing for Social Debiasing

Large Language Model Bias Mitigation from the Perspective of Knowledge Editing

Debiasing Multimodal Large Language Models

Debias your Large Multi-Modal Model at Test-Time with Non-Contrastive Visual Attribute Steering

A Multi-LLM Debiasing Framework

Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective

MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency

Social Debiasing for Fair Multi-modal LLMs

Mitigating Gender Bias in Code Large Language Models via Model Editing

Locating and Mitigating Gender Bias in Large Language Models

Editing Large Language Models: Problems, Methods, and Opportunities

Language Anisotropic Cross-Lingual Model Editing

Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue

Model Editing Can Hurt General Abilities of Large Language Models

Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework

More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Models

BCD-MM: Multimodal Sentiment Analysis Model With Dual-Bias-Aware Feature Learning and Attention Mechanisms

Steering LLMs Towards Unbiased Responses: A Causality-Guided Debiasing Framework

Untying the Reversal Curse via Bidirectional Language Model Editing