Abstract:Document Visual Question Answering (VQA) requires models to interpret textual information within complex visual layouts and comprehend spatial relationships to answer questions based on document images. Existing approaches often lack interpretability and fail to precisely localize answers within the document, hindering users' ability to verify responses and understand the reasoning process. Moreover, standard metrics like Average Normalized Levenshtein Similarity (ANLS) focus on text accuracy but overlook spatial correctness. We introduce DLaVA, a novel method that enhances Multimodal Large Language Models (MLLMs) with answer localization capabilities for Document VQA. Our approach integrates image annotation directly into the MLLM pipeline, improving interpretability by enabling users to trace the model's reasoning. We present both OCR-dependent and OCR-free architectures, with the OCR-free approach eliminating the need for separate text recognition components, thus reducing complexity. To the best of our knowledge, DLaVA is the first approach to introduce answer localization within multimodal QA, marking a significant step forward in enhancing user trust and reducing the risk of AI hallucinations. Our contributions include enhancing interpretability and reliability by grounding responses in spatially annotated visual content, introducing answer localization in MLLMs, proposing a streamlined pipeline that combines an MLLM with a text detection module, and conducting comprehensive evaluations using both textual and spatial accuracy metrics, including Intersection over Union (IoU). Experimental results on standard datasets demonstrate that DLaVA achieves SOTA performance, significantly enhancing model transparency and reliability. Our approach sets a new benchmark for Document VQA, highlighting the critical importance of precise answer localization and model interpretability.

What problem does this paper attempt to address?

This paper attempts to solve several key problems in the Document Visual Question Answering (DocVQA) task: 1. **Lack of interpretability and answer precise - location ability**: - When dealing with DocVQA, existing methods usually cannot provide sufficient interpretability and it is difficult to precisely locate the answers in the document image. This makes it difficult for users to verify the model's responses and understand its reasoning process. 2. **Limitations of standard evaluation metrics**: - Commonly used evaluation metrics such as Average Normalized Levenshtein Similarity (ANLS) only focus on text accuracy and ignore spatial correctness. Spatial evaluation metrics such as Intersection over Union (IoU) are also often overlooked. 3. **Complexity and error propagation**: - Methods relying on Optical Character Recognition (OCR) increase the complexity of the system and may lead to error propagation, affecting the overall performance. To solve these problems, the author introduced a new method - **DLaVA** (Document Language and Vision Assistant), which enhances the answer - location ability of multi - modal large language models (MLLMs) in DocVQA. Specifically, the main contributions of DLaVA include: - **Improving interpretability and reliability**: By combining responses with spatially - annotated visual content, it enhances users' trust and reduces the risk of AI hallucination. - **Introducing answer - location function**: For the first time, the answer - location function is introduced in MLLMs, filling an important gap in current DocVQA methods. - **Innovative pipeline design and model analysis**: A simplified pipeline is proposed, which combines MLLM with a text - detection module, eliminating the need for an external OCR component and reducing complexity. - **Comprehensive evaluation**: Using dual evaluation metrics of text and spatial accuracy (including IoU), the model performance is thoroughly evaluated. Through these improvements, DLaVA not only improves the transparency and reliability of the model but also shows excellent performance on multiple standard datasets, setting a new benchmark in the DocVQA field.

DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness

Simple and Effective Visual Question Answering in a Single Modality

Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering

R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest

Multimodal attention-driven visual question answering for Malayalam

STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering

LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound

Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts

Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs

Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery

Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks

Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering

Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation

LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description

Localized Questions in Medical Visual Question Answering

LOVA3: Learning to Visual Question Answering, Asking and Assessment

From Images to Textual Prompts: Zero-Shot Visual Question Answering with Frozen Large Language Models

Improving Automatic VQA Evaluation Using Large Language Models

PDF-MVQA: A Dataset for Multimodal Information Retrieval in PDF-based Visual Question Answering

MGA-VQA: Multi-Granularity Alignment for Visual Question Answering

Consistency-preserving Visual Question Answering in Medical Imaging