Abstract:Multimodal Question Answering (MMQA) has emerged as a challenging frontier at the intersection of natural language processing (NLP) and computer vision, demanding the integration of diverse modalities for effective comprehension and response. While pre-trained language models (PLMs) exhibit impressive performance across a range of NLP tasks, the investigation of text-based approaches to address MMQA represents a compelling and promising avenue for further research and advancement in the field. Although recent research has delved into text-based approaches for MMQA, the attained results have been unsatisfactory, which could be attributed to potential information loss during the knowledge transformation processes. In response, a novel three-stage framework named UniRaG is proposed for tackling MMQA, which encompasses unified knowledge representation, context retrieval, and answer generation. At the initial stage, advanced techniques are employed for unified knowledge representation, including LLaVA for image captioning and table linearization for tabular data, facilitating seamless integration of visual and tabular information into textual representation. For context retrieval, a cross-encoder trained on sequence classification is utilized to predict relevance scores for question-document pairs, and a top-k retrieval strategy is employed to retrieve the documents with the highest relevance scores as the contexts for answer generation. Finally, the answer generation stage is facilitated by a text-to-text PLM, Flan-T5-Base, which follows the encoder-decoder architecture with attention mechanisms. During this stage, uniform prefix conditioning is applied to the input text for enhanced adaptability and generalizability. Moreover, contextual diversity training is introduced to improve model robustness by including distractor documents as negative contexts during training. Experimental results on the MultimodalQA dataset demonstrate the superior performance of UniRaG, surpassing the existing state-of-the-art methods across all scenarios with 67.4% EM and 71.3% F1. Overall, UniRaG showcases robustness and reliability in MMQA, heralding significant advancements in multimodal comprehension and question answering research.

Multimodal Question Answering for Unified Information Extraction

Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

An Interactive Multi-modal Query Answering System with Retrieval-Augmented Large Language Models

UniRaG: Unification, Retrieval, and Generation for Multimodal Question Answering With Pre-Trained Language Models

UMIE: Unified Multimodal Information Extraction with Instruction Tuning

Modality attention fusion model with hybrid multi-head self-attention for video understanding

MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering

An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism

MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering

3D Question Answering for City Scene Understanding

PokeMQA: Programmable knowledge editing for Multi-hop Question Answering

Multi-modal Situated Reasoning in 3D Scenes

Synthetic Multimodal Question Generation

UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing

Bridging the Language Gap: Knowledge Injected Multilingual Question Answering

MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

Multifaceted Improvements for Conversational Open-Domain Question Answering

Modular Blended Attention Network for Video Question Answering