Abstract:We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities: text, images, and tables. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs. Our questions are ambiguous, in that the modality that contains the answer is not easily determined based solely upon the question. To demonstrate this ambiguity, we construct a modality selector (or disambiguator) network, and this model gets substantially lower accuracy on our challenge set, compared to existing datasets, indicating that our questions are more ambiguous. By analyzing this model, we investigate which words in the question are indicative of the modality. Next, we construct a simple baseline ManyModalQA model, which, based on the prediction from the modality selector, fires a corresponding pre-trained state-of-the-art unimodal QA model. We focus on providing the community with a new manymodal evaluation set and only provide a fine-tuning set, with the expectation that existing datasets and approaches will be transferred for most of the training, to encourage low-resource generalization without large, monolithic training sets for each new task. There is a significant gap between our baseline models and human performance; therefore, we hope that this challenge encourages research in end-to-end modality disambiguation and multimodal QA models, as well as transfer learning. Code and data available at: <a class="link-external link-https" href="https://github.com/hannandarryl/ManyModalQA" rel="external noopener nofollow">this https URL</a>

Multimodal Question Answering over Structured Data with Ambiguous Entities

Entity-Relation Extraction As Multi-Turn Question Answering

Enhancing Multi-modal Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation

Multimodal Question Answering for Unified Information Extraction

Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation

An Interactive Multi-modal Query Answering System with Retrieval-Augmented Large Language Models

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search

DIEM: Decomposition-Integration Enhancing Multimodal Insights

ManyModalQA: Modality Disambiguation and QA over Diverse Inputs

Discovering Multimodal Hierarchical Structures with Graph Neural Networks for Multi-modal and Multi-hop Question Answering.

AmbigDocs: Reasoning across Documents on Different Entities under the Same Name

Query-Driven Knowledge Graph Construction using Question Answering and Multimodal Fusion

Multi-Task Learning for Conversational Question Answering over a Large-Scale Knowledge Base

An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism

OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering

Cross-modal Retrieval for Knowledge-based Visual Question Answering

Research on Dual-Dimensional Entity Association-Based Question and Answering Technology for Smart Medicine

Exploiting Hybrid Semantics of Relation Paths for Multi-hop Question Answering Over Knowledge Graphs

Entailment-based Question Answering for Structured Data.