Abstract:Multi-modal knowledge graph (MKG) refers to a structured semantic network that accurately represents the real-world information by incorporating multiple modalities. Existing researches primarily focus on leveraging multi-modal fusion to enhance the representation capability of entity nodes and link prediction to deal with the incompleteness of the MKG. However, the inherent heterogeneity between structural modality and semantic modality poses challenges to the multi-modal fusion, as noise interference could compromise the effectiveness of the fusion representation. In this study, we propose a novel hierarchical Transformer architecture, named MNFormer, which captures the structural and semantic information while avoiding heterogeneity issues by fully integrating both multi-hop neighbor paths and image-text embeddings. During the encoding stage of MNFormer, we design multiple layers of Multi-hop Neighbor Fusion (MNF) module that employ attentions to merge the image and text features. These MNF modules progressively fuse the information of neighboring entities hop by hop along the neighbor paths of the source entity. The Transformer during decoding stage is then utilized to integrate the outputs of all MNF modules, whose output is subsequently employed to match target entities and accomplish MKG completion. Moreover, we develop a semantic direction loss to enhance the fitting performance of MNFormer. Experimental results on four datasets demonstrate that MNFormer exhibits notable competitiveness when compared to the state-of-the-art models. Additionally, ablation studies showcase the significant ability of MNFormer to effectively combine structural and semantic information, leading to enhanced performance through complementary enhancements.

Multi-tensor Completion with Common Structures.

MFF-Net: Towards Efficient Monocular Depth Completion With Multi-Modal Feature Fusion

Up-to-Down Network: Fusing Multi-Scale Context for 3D Semantic Scene Completion

Multi-Dimensional Visual Data Completion via Low-Rank Tensor Representation Under Coupled Transform

Multi-Tensor Network Representation for High-Order Tensor Completion

Visual Data Completion Via Local Sensitive Low Rank Tensor Learning.

A Biased Deep Tensor Factorization Network For Tensor Completion

Structure-Aware Cross-Modal Transformer for Depth Completion

Tensor Completion Via Collaborative Sparse and Low-Rank Transforms

Adaptive Context-Aware Multi-Modal Network for Depth Completion

MLCTR: A Fast Scalable Coupled Tensor Completion Based on Multi-Layer Non-Linear Matrix Factorization

Tensor Completion via Complementary Global, Local, and Nonlocal Priors

Tensor completion via multi-directional partial tensor nuclear norm with total variation regularization

Multi-hop neighbor fusion enhanced hierarchical transformer for multi-modal knowledge graph completion

A Novel Truncated Normal Tensor Completion Method for Multi-Source Fusion Data

MFM-Net: Unpaired Shape Completion Network with Multi-stage Feature Matching.

Adaptive Graph Completion Based Incomplete Multi-View Clustering

Tensor Completion Algorithms in Big Data Analytics

Beyond Low Rank: A Graph-Based Propagation Approach to Tensor Completion for Multi-Acquisition Scenarios

Deep Sparse Depth Completion Using Multi-Affinity Matrix

MLSFF: Multi-level structural features fusion for multi-modal knowledge graph completion