Abstract:Knowledge based visual question-answering is an emerging technique that combines computer vision and natural language processing to address image-based questions. This approach requires the model to possess internal reasoning ability and incorporate external knowledge to enhance its generalization performance. Knowledge graphs are commonly employed to integrate various types of knowledge, such as image knowledge, text knowledge, and basic common sense, into visual language question answering models, significantly enhancing their interpretability. However, introducing knowledge into these models often involves retrieval and pre-training methods, which can introduce noise due to the local correlation among knowledge sources and the existence of modality gaps, thereby affecting the model's performance. To mitigate this issue, this paper proposes a multimodal knowledge extraction approach based on distributed representation learning. The approach models inexpressible multimodal facts using explicit triples, considering the semantic gap between visual and textual modalities. Cross-modal representations are obtained through an attention mechanism, resulting in knowledge triplets comprising visual features, cross-modal representations, and text features. By incorporating these knowledge triplets into the visual language question answering model, the task is transformed into pattern matching using knowledge triples. The proposed approach comprehensively considers multiple factors, is not restricted to specific forms of knowledge, and can effectively incorporate a substantial amount of knowledge. Experimental results demonstrate its superiority, with a performance improvement of 0.5% on the OKVQA dataset compared to the baseline model, highlighting its strong generalization ability.

Knowledge representation learning method in combination with entity images and server

Knowledge Representation Learning with Entity Descriptions, Hierarchical Types, and Textual Relations

Knowledge graph representation learning method and system in combination with entity description

Image-embodied Knowledge Representation Learning.

Knowledge graph representation learning method through combination of entity hierarchy category

A Representation Learning Method of Knowledge Graph Integrating Ordered Relation Path and Entity Description Information

Representation Learning of Knowledge Graphs with Entity Descriptions

A Representation Learning Method of Knowledge Graph Integrating Relationship Path, Text Description and Entity Hierarchy Type Information

Distributed Representation Learning for Knowledge Graphs with Entity Descriptions

Representation Learning with Entity Topics for Knowledge Graphs.

Knowledge Representation Learning with Entities, Attributes and Relations.

General Knowledge Embedded Image Representation Learning

Improving Knowledge Graph Representation Learning by Structure Contextual Pre-training

Word vector learning model training method and server

A survey: knowledge graph entity alignment research based on graph embedding

ASRC:A Knowledge Graph Relation Construction Model based on Active Learning and Semantic Recognition

SG-CIM Mapping Entity Relationship Inference and Verification Techniques Based on Graph Representation Learning Technology

Research on Joint Representation Learning Methods for Entity Neighborhood Information and Description Information

Knowledge Representation Via Joint Learning of Sequential Text and Knowledge Graphs.

Building method and system used for knowledge obtaining model in knowledge graph

Multimodal Knowledge Triple Extraction Based on Representation Learning