Abstract:Knowledge graph embedding models have become a popular approach for knowledge graph completion through predicting the plausibility of (potential) triples. This is performed by transforming the entities and relations of the knowledge graph into an embedding space. However, knowledge graphs often include further textual information stored in literal, which is ignored by such embedding models. As a consequence, the learning process stays limited to the structure and the connections between the entities, which has the potential to negatively influence the performance. We bridge this gap by leveraging the capabilities of pre-trained language models to include textual knowledge in the learning process of embedding models. This is achieved by introducing a new loss function that guides embedding models in measuring the likelihood of triples by taking such complementary knowledge into consideration. The proposed solution is a model-independent loss function that can be plugged into any knowledge graph embedding model. In this paper, Sentence-BERT and fastText are used as pre-trained language models from which the embeddings of the textual knowledge are obtained and injected into the loss function. The loss function contains a trainable slack variable that determines the degree to which the language models influence the plausibility of triples. Our experimental evaluation on six benchmarks, namely Nations, UMLS, WordNet, and three versions of CodEx confirms the advantage of using pre-trained language models for boosting the accuracy of knowledge graph embedding models. We showcase this by performing evaluations on top of the five well-known knowledge graph embedding models such as TransE, RotatE, ComplEx, DistMult, and QuatE. The results show an improvement in accuracy up to 9% on UMLS dataset for the Distmult model and 4.2% on the Nations dataset for the ComplEx model when they are guided by pre-trained language models. We additionally- studied the effect of multiple factors such as the structure of the knowledge graphs and training steps and presented them as ablation studies.

Improved Sequence Predictions using Knowledge Graph Embedding for Large Language Models

Large Language Models and Knowledge Graphs: Opportunities and Challenges

Knowledge Graph-Enhanced Large Language Models via Path Selection

Combining Knowledge Graphs and Large Language Models

Knowledge Graph Large Language Model (KG-LLM) for Link Prediction

Supervised Knowledge Makes Large Language Models Better In-context Learners

Exploring Large Language Models for Knowledge Graph Completion

Large Language Model Enhanced Knowledge Representation Learning: A Survey

Enhancing Knowledge Graph Construction Using Large Language Models

LPNL: Scalable Link Prediction with Large Language Models

Large Language Models on Graphs: A Comprehensive Survey

Distilling Event Sequence Knowledge From Large Language Models

Relations Prediction for Knowledge Graph Completion using Large Language Models

Synergizing Knowledge Graphs with Large Language Models: A Comprehensive Review and Future Prospects

Large Language Models as Interpolated and Extrapolated Event Predictors

Language Models as Knowledge Embeddings

Large Language Models Can Better Understand Knowledge Graphs Than We Thought

Empowering Small-Scale Knowledge Graphs: A Strategy of Leveraging General-Purpose Knowledge Graphs for Enriched Embeddings

Language Model Guided Knowledge Graph Embeddings

EmbedLLM: Learning Compact Representations of Large Language Models