Abstract:Multimodal Large Language Models (MLLMs) have displayed remarkable performance in multi-modal tasks, particularly in visual comprehension. However, we reveal that MLLMs often generate incorrect answers even when they understand the visual content. To this end, we manually construct a benchmark with 12 categories and design evaluation metrics that assess the degree of error in MLLM responses even when the visual content is seemingly understood. Based on this benchmark, we test 15 leading MLLMs and analyze the distribution of attention maps and logits of some MLLMs. Our investigation identifies two primary issues: 1) most instruction tuning datasets predominantly feature questions that 'directly' relate to the visual content, leading to a bias in MLLMs' responses to other indirect questions, and 2) MLLMs' attention to visual tokens is notably lower than to system and question tokens. We further observe that attention scores between questions and visual tokens as well as the model's confidence in the answers are lower in response to misleading questions than to straightforward ones. To address the first challenge, we introduce a paired positive and negative data construction pipeline to diversify the dataset. For the second challenge, we propose to enhance the model's focus on visual content during decoding by refining the text and visual prompt. For the text prompt, we propose a content guided refinement strategy that performs preliminary visual content analysis to generate structured information before answering the question. Additionally, we employ a visual attention refinement strategy that highlights question-relevant visual tokens to increase the model's attention to visual content that aligns with the question. Extensive experiments demonstrate that these challenges can be significantly mitigated with our proposed dataset and techniques.

Probing Multimodal Large Language Models for Global and Local Semantic Representations

A Survey of Multimodal Large Language Model from A Data-centric Perspective

Explaining Multi-modal Large Language Models by Analyzing their Vision Perception

MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model

A Survey on Multimodal Large Language Models

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey

From Image to Video, what do we need in multimodal LLMs?

MR-MLLM: Mutual Reinforcement of Multimodal Comprehension and Vision Perception

Multimodal Large Language Models: A Survey

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly

Exploring Graph Structure Comprehension Ability of Multimodal Large Language Models: Case Studies

A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models

Efficient Multimodal Large Language Models: A Survey

A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks

Demonstrative Instruction Following in Multimodal LLMs Via Integrating Low-Rank Adaptation with Ensemble Learning

Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models