Modality-Aware Negative Sampling for Multi-modal Knowledge Graph Embedding

Yichi Zhang,Mingyang Chen,Wen Zhang

2023-04-23

Abstract:Negative sampling (NS) is widely used in knowledge graph embedding (KGE), which aims to generate negative triples to make a positive-negative contrast during training. However, existing NS methods are unsuitable when multi-modal information is considered in KGE models. They are also inefficient due to their complex design. In this paper, we propose Modality-Aware Negative Sampling (MANS) for multi-modal knowledge graph embedding (MMKGE) to address the mentioned problems. MANS could align structural and visual embeddings for entities in KGs and learn meaningful embeddings to perform better in multi-modal KGE while keeping lightweight and efficient. Empirical results on two benchmarks demonstrate that MANS outperforms existing NS methods. Meanwhile, we make further explorations about MANS to confirm its effectiveness.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the issues of inapplicability and inefficiency of negative sampling (NS) methods in multimodal knowledge graph embedding (MMKGE). Existing negative sampling methods are designed for general knowledge graph embedding (KGE) models and perform poorly when dealing with multimodal knowledge graphs that contain rich modal information such as images and text. Additionally, these methods incur high computational costs due to their complex design. Therefore, the paper proposes a Modality-Aware Negative Sampling (MANS) strategy to address the aforementioned issues, enabling multimodal knowledge graph embedding models to better align structural and visual embeddings while learning more meaningful embeddings in a lightweight and efficient manner.

Specifically, the main contributions of the paper include:
1. Proposing MANS, the first negative sampling strategy focused on multimodal knowledge graph embedding.
2. Introducing MANS-V (Visual Negative Sampling) within MANS to align information from different modalities. Further extending MANS-V, three combined negative sampling strategies are proposed: Two-Stage Negative Sampling (MANS-T), Hybrid Negative Sampling (MANS-H), and Adaptive Negative Sampling (MANS-A).
3. Validating the superior performance of MANS in various tasks through experiments on two multimodal knowledge graph datasets.
4. Conducting extensive analysis to explore several research questions regarding MANS, demonstrating its effectiveness and rationality.

By proposing MANS, the paper not only addresses the shortcomings of existing negative sampling methods in multimodal scenarios but also provides a lightweight and efficient solution, thereby improving the performance of multimodal knowledge graph embedding models.