Multimodal Analogical Reasoning over Knowledge Graphs

Ningyu Zhang,Lei Li,Xiang Chen,Xiaozhuan Liang,Shumin Deng,Huajun Chen

DOI: https://doi.org/10.48550/arXiv.2210.00312

2023-03-01

Abstract:Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources. To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task. We further propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory, which can obtain better performance. Code and datasets are available in <a class="link-external link-https" href="https://github.com/zjunlp/MKG_Analogy" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning,Multimedia

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the ability to perform multi - modal analogical reasoning on knowledge graphs. Specifically, existing research mainly focuses on single - modal analogical reasoning and ignores the utilization of structural knowledge. However, research in cognitive psychology shows that information from multi - modal sources can bring stronger cognitive transfer abilities than single - modal ones. Therefore, this paper proposes a new task - multi - modal analogical reasoning on knowledge graphs, which requires the model to have the ability to perform multi - modal reasoning using background knowledge.

To achieve this goal, the author constructs a multi - modal analogical reasoning dataset (MARS) and a multi - modal knowledge graph (MarKG). By evaluating multi - modal knowledge graph embeddings and pre - trained Transformer baseline models, the potential challenges of the proposed task are demonstrated. In addition, the author also proposes a Transformer - based multi - modal analogical reasoning framework (MarT), which is inspired by the structural mapping theory and can achieve better performance.

In general, the main contributions of this paper include:
1. Introducing a new multi - modal analogical reasoning task, advancing the traditional analogical learning settings.
2. Constructing a dataset MARS containing a multi - modal knowledge graph, providing support for studying the multi - modal analogical reasoning abilities of neural networks.
3. Reporting the performance of various multi - modal knowledge graph embeddings, multi - modal pre - trained Transformer baselines, and the proposed MarT framework, and discussing the potential of this task, hoping to promote future research on zero - shot learning and domain generalization in the fields of computer vision and natural language processing.