Abstract:The field of visual question answering (VQA) has seen a growing trend of integrating external knowledge sources to improve performance. However, owing to the potential incompleteness of external knowledge sources and the inherent mismatch between different forms of data, current knowledge-based visual question answering (KBVQA) techniques are still confronted with the challenge of effectively integrating and utilizing multiple heterogeneous data. To address this issue, a novel approach centered on a multi-modal semantic graph (MSG) is proposed. The MSG serves as a mechanism for effectively unifying the representation of heterogeneous data and diverse types of knowledge. Additionally, a multi-modal semantic graph knowledge reasoning model (MSG-KRM) is introduced to perform reasoning and deep fusion of image–text information and external knowledge sources. The development of the semantic graph involves extracting keywords from the image object detection information, question text, and external knowledge texts, which are then represented as symbol nodes. Three types of semantic graphs are then constructed based on the knowledge graph, including vision, question, and the external knowledge text, with non-symbol nodes added to connect these three independent graphs and marked with respective node and edge types. During the inference stage, the multi-modal semantic graph and image–text information are embedded into the feature semantic graph through three embedding methods, and a type-aware graph attention module is employed for deep reasoning. The final answer prediction is a blend of the output from the pre-trained model, graph pooling results, and the characteristics of non-symbolic nodes. The experimental results on the OK-VQA dataset show that the MSG-KRM model is superior to existing methods in terms of overall accuracy score, achieving a score of 43.58, and with improved accuracy for most subclass questions, proving the effectiveness of the proposed method.

Answering Knowledge-Based Visual Questions Via the Exploration of Question Purpose

Simple and Effective Visual Question Answering in a Single Modality

Perceptual Visual Reasoning with Knowledge Propagation

Knowledge-Augmented Visual Question Answering With Natural Language Explanation

Knowledge-Enhanced Visual Question Answering with Multi-modal Joint Guidance.

Open-Set Knowledge-Based Visual Question Answering with Inference Paths

Knowledge-aware image understanding with multi-level visual representation enhancement for visual question answering

Boosting Visual Question Answering with Context-aware Knowledge Aggregation

Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Explicit Knowledge-based Reasoning for Visual Question Answering

Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts

Question-Guided Semantic Dual-Graph Visual Reasoning with Novel Answers.

Proposing Plausible Answers for Open-ended Visual Question Answering

Can Open Domain Question Answering Systems Answer Visual Knowledge Questions?

Right this way: Can VLMs Guide Us to See More to Answer Questions?

Q&A Prompts: Discovering Rich Visual Clues through Mining Question-Answer Prompts for VQA requiring Diverse World Knowledge

Knowledge-Based Visual Question Answering Using Multi-Modal Semantic Graph

Video Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

Video Question Answering With Prior Knowledge and Object-Sensitive Learning