Abstract:A vast amount of textual and structural information is required for knowledge graph construction and its downstream tasks. However, most of the current knowledge graphs are incomplete due to the difficulty of knowledge acquisition and integration. Knowledge Graph Completion (KGC) is used to predict missing connections. In previous studies, textual information and graph structural information are utilized independently, without an effective method for fusing these two types of information. In this paper, we propose a graph structure enhanced pre-training language model for knowledge graph completion. Firstly, we design a graph sampling algorithm and a Graph2Seq module for constructing sub-graphs and their corresponding contexts to support large-scale knowledge graph learning and parallel training. It is also the basis for fusing textual data and graph structure. Next, two pre-training tasks based on masked modeling are designed for capturing accurate entity-level and relation-level information. Furthermore, this paper proposes a novel asymmetric Encoder-Decoder architecture to restore masked components, where the encoder is a Pre-trained Language Model (PLM) and the decoder is a multi-relational Graph Neural Network (GNN). The purpose of the architecture is to integrate textual information effectively with graph structural information. Finally, the model is fine-tuned for KGC tasks on two widely used public datasets. The experiments show that the model achieves excellent performance and outperforms baselines in most metrics, which demonstrate the effectiveness of our approach by fusing the structure and semantic information to knowledge graph.

PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion

KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective

Pre-training Transformers for Knowledge Graph Completion

Knowledge-Infused Pre-trained Models for KG Completion

Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer.

Augmented Large Language Models with Parametric Knowledge Guiding

KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion

Multilingual Knowledge Graph Completion from Pretrained Language Models with Knowledge Constraints

Enhanced Transfer Learning with Efficient Modeling and Adaptive Fusion of Knowledge Via Prompt Tuning

Unlock the Power of Frozen LLMs in Knowledge Graph Completion

LLM-Neo: Parameter Efficient Knowledge Distillation for Large Language Models

Graph Structure Enhanced Pre-Training Language Model for Knowledge Graph Completion

Decouple Knowledge from Paramters for Plug-and-play Language Modeling.

Simplified Representation Learning Model Based On Parameter-Sharing For Knowledge Graph Completion

Towards a Unified View of Parameter-Efficient Transfer Learning

Making Large Language Models Perform Better in Knowledge Graph Completion

CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models