Abstract:Transformers have achieved remarkable performance in widespread fields, including natural language processing, computer vision and graph mining. However, vanilla Transformer architectures have not yielded promising improvements in the Knowledge Graph (KG) representations, where the translational distance paradigm dominates this area. Note that vanilla Transformer architectures struggle to capture the intrinsically heterogeneous structural and semantic information of knowledge graphs. To this end, we propose a new variant of Transformer for knowledge graph representations dubbed Relphormer. Specifically, we introduce Triple2Seq which can dynamically sample contextualized sub-graph sequences as the input to alleviate the heterogeneity issue. We propose a novel structure-enhanced self-attention mechanism to encode the relational information and keep the semantic information within entities and relations. Moreover, we utilize masked knowledge modeling for general knowledge graph representation learning, which can be applied to various KG-based tasks including knowledge graph completion, question answering, and recommendation. Experimental results on six datasets show that Relphormer can obtain better performance compared with baselines. Code is available in <a class="link-external link-https" href="https://github.com/zjunlp/Relphormer" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main goal of this paper is to propose a new Transformer-based method, called Relphormer (Relational Graph Transformer), to address several key challenges in Knowledge Graph (KG) representation learning and improve performance on various KG-based tasks. Specifically, the paper aims to address the following issues: 1. **Addressing the heterogeneity of knowledge graphs**: Knowledge graphs consist of different types of relationships that carry different semantic information, leading to heterogeneity in representations. Traditional Transformer architectures typically consider only single-hop neighbors, ignoring the inherent complex structures in knowledge graphs. 2. **Integrating topological structure and textual descriptions**: Each node in a knowledge graph has two types of knowledge—topological structure and textual descriptions. Different nodes have unique topological and textual features. Previous works often treat entities and relationships as ordinary tokens, neglecting important structural information. 3. **Unified optimization objective**: Most existing methods use predefined scoring functions to optimize knowledge embeddings. However, this approach requires different optimization objectives for entity/relationship prediction and involves expensive scoring of all possible triples during inference. Additionally, existing knowledge graph representation learning methods struggle to provide a unified learning representation for various KG-based tasks. To tackle these challenges, the paper proposes the following key techniques: - **Triple2Seq**: This is a dynamic sampling strategy used to extract local contextual subgraphs from the knowledge graph as input sequences to alleviate the heterogeneity issue. - **Structure-enhanced self-attention mechanism**: This mechanism can encode textual features while preserving structural information simultaneously. - **Masked knowledge modeling**: This is a unified optimization method for predicting masked entities and relationship labels in the input sequence, thereby simplifying the optimization process for entity and relationship prediction tasks. Through the above techniques, Relphormer aims to improve the expressive power of existing knowledge graph representation learning methods and provide a unified representation framework suitable for various KG-based tasks, such as knowledge graph completion, question answering, and recommendation. Experimental results show that Relphormer achieves better performance on multiple benchmark datasets.

Relphormer: Relational Graph Transformer for Knowledge Graph Representations

Position-Aware Relational Transformer for Knowledge Graph Embedding.

TGformer: A Graph Transformer Framework for Knowledge Graph Embedding

Structure Pre-training and Prompt Tuning for Knowledge Graph Transfer

Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries

Representation Learning on Hyper-Relational and Numeric Knowledge Graphs with Transformers

Transition-based Knowledge Graph Embedding with Relational Mapping Properties

KnowFormer: Revisiting Transformers for Knowledge Graph Reasoning

Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion

MTKGCformer: A Multi-train Transformer-based Representation Learning for Knowledge Graph Completion Task

DT4KGR: Decision transformer for fast and effective multi-hop reasoning over knowledge graphs

Neighbor-T: Neighborhood Transformer Aggregation for Enhancing Representation of Out-of-knowledge-base Entities

TripleRE: Knowledge Graph Embeddings via Tripled Relation Vectors

Leveraging Knowledge Graph for Domain-Specific Chinese Named Entity Recognition Via Lexicon-Based Relational Graph Transformer

Gophormer: Ego-Graph Transformer for Node Classification

Relation classification via knowledge graph enhanced transformer encoder

Knowledge Graph Embedding for Hyper-Relational Data

RelTransformer: A Transformer-Based Long-Tail Visual Relationship Recognition

MRE: A Translational Knowledge Graph Completion Model Based on Multiple Relation Embedding

TransAM: Transformer appending matcher for few-shot knowledge graph completion

Text-Enhanced Representation Learning for Knowledge Graph.