Abstract:The field of visual question answering (VQA) has seen a growing trend of integrating external knowledge sources to improve performance. However, owing to the potential incompleteness of external knowledge sources and the inherent mismatch between different forms of data, current knowledge-based visual question answering (KBVQA) techniques are still confronted with the challenge of effectively integrating and utilizing multiple heterogeneous data. To address this issue, a novel approach centered on a multi-modal semantic graph (MSG) is proposed. The MSG serves as a mechanism for effectively unifying the representation of heterogeneous data and diverse types of knowledge. Additionally, a multi-modal semantic graph knowledge reasoning model (MSG-KRM) is introduced to perform reasoning and deep fusion of image–text information and external knowledge sources. The development of the semantic graph involves extracting keywords from the image object detection information, question text, and external knowledge texts, which are then represented as symbol nodes. Three types of semantic graphs are then constructed based on the knowledge graph, including vision, question, and the external knowledge text, with non-symbol nodes added to connect these three independent graphs and marked with respective node and edge types. During the inference stage, the multi-modal semantic graph and image–text information are embedded into the feature semantic graph through three embedding methods, and a type-aware graph attention module is employed for deep reasoning. The final answer prediction is a blend of the output from the pre-trained model, graph pooling results, and the characteristics of non-symbolic nodes. The experimental results on the OK-VQA dataset show that the MSG-KRM model is superior to existing methods in terms of overall accuracy score, achieving a score of 43.58, and with improved accuracy for most subclass questions, proving the effectiveness of the proposed method.

BERT-based Semantic Query Graph Extraction for Knowledge Graph Question Answering

Knowledge Graph Question Answering with semantic oriented fusion model.

Semantic Structure based Query Graph Prediction for Question Answering over Knowledge Graph

Semantic Parsing for Question Answering over Knowledge Graphs

Neural-Symbolic Entangled Framework for Complex Query Answering

Outlining and Filling: Hierarchical Query Graph Generation for Answering Complex Questions over Knowledge Graphs

Leveraging Knowledge Graph Embeddings For Natural Language Question Answering

Simple Question Answering over Knowledge Graph Enhanced by Question Pattern Classification

Formal Query Generation for Question Answering over Knowledge Bases

Tree-KGQA: An Unsupervised Approach for Question Answering Over Knowledge Graphs

BERT and hierarchical cross attention-based question answering over bridge inspection knowledge graph

UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph

Knowledge Graph Question Answering via SPARQL Silhouette Generation

Gs-Cbr-Kbqa: Graph-Structured Case-Based Reasoning for Knowledge Base Question Answering

Knowledge-Based Visual Question Answering Using Multi-Modal Semantic Graph

Knowledge-Enhanced Retrieval: A Scheme for Question Answering

Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answering

A BERT-Bi-LSTM-Based Knowledge Graph Question Answering Method

Svega: Answering Natural Language Questions over Knowledge Base with Semantic Matching

Improving Question Answering over Knowledge Graphs Using Graph Summarization