Abstract:Extrapolation in Large language models (LLMs) for open-ended inquiry encounters two pivotal issues: (1) hallucination and (2) expensive training costs. These issues present challenges for LLMs in specialized domains and personalized data, requiring truthful responses and low fine-tuning costs. Existing works attempt to tackle the problem by augmenting the input of a smaller language model with information from a knowledge graph (KG). However, they have two limitations: (1) failing to extract relevant information from a large one-hop neighborhood in KG and (2) applying the same augmentation strategy for KGs with different characteristics that may result in low performance. Moreover, open-ended inquiry typically yields multiple responses, further complicating extrapolation. We propose a new task, the extreme multi-label KG link prediction task, to enable a model to perform extrapolation with multiple responses using structured real-world knowledge. Our retriever identifies relevant one-hop neighbors by considering entity, relation, and textual data together. Our experiments demonstrate that (1) KGs with different characteristics require different augmenting strategies, and (2) augmenting the language model's input with textual data improves task performance significantly. By incorporating the retrieval-augmented framework with KG, our framework, with a small parameter size, is able to extrapolate based on a given KG. The code can be obtained on GitHub:

What problem does this paper attempt to address?

The paper primarily focuses on addressing two core issues encountered by Large Language Models (LLMs) in open-ended queries: hallucination and the expensive cost of training. These issues are particularly prominent in professional domains and applications involving personalized data, as these scenarios require the model to provide highly accurate and truthful responses, along with lower fine-tuning costs. Existing work attempts to mitigate these problems by incorporating Knowledge Graph (KG) information into smaller language model inputs, but there are limitations, including the inability to effectively extract relevant information from the one-hop neighborhood of the KG, and the application of the same enhancement strategy to different types of KGs leading to poor performance. The paper introduces a new task—Extreme Multi-Label Knowledge Graph Link Prediction—aimed at enabling models to extrapolate multiple responses using structured real-world knowledge. Researchers designed a retrieval-enhanced framework that can identify relevant one-hop neighbors involving entities, relationships, and textual data. Experiments show that different types of KGs indeed require different enhancement strategies, and adding textual data to the language model's input can significantly improve task performance. By combining the retrieval-enhanced framework with KGs, models can extrapolate based on the given KG, even with a smaller number of parameters. Specifically, the paper addresses the following key challenges: 1. Handling cases where the one-hop neighborhood of nodes in the KG may be very large, ensuring the model can extract the most relevant information. 2. Adopting different enhancement strategies for KGs with varying characteristics to optimize model performance. 3. Addressing the issue that open-ended queries often generate multiple responses, making extrapolation more complex. 4. Maximizing the number of triples in each fixed-length raw text input under limited input length by truncating descriptions without changing the original meaning. 5. Addressing the issue of low precision that may occur in high-dimensional classification layers when optimizing with binary cross-entropy loss, proposing a new loss function and training strategy. Experimental results prove that the proposed framework and methods show significant improvements in the Precision at k metric when dealing with large knowledge graphs such as WN18RR and FB15k-237, compared to other models. Additionally, through a series of ablation experiments, the paper validates the importance of descriptive data, the SimCSE model, and the filtered 1-hop neighbor selection strategy for model performance. Lastly, the paper discusses the different values of the positive coefficient α in the loss function, proving that its reasonable setting at different stages is crucial for balancing the model's recall and precision.

Retrieval-Augmented Language Model for Extreme Multi-Label Knowledge Graph Link Prediction

Knowledge Graph Large Language Model (KG-LLM) for Link Prediction

Knowledge Graph-Enhanced Large Language Models via Path Selection

Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs

Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation

KG-Retriever: Efficient Knowledge Indexing for Retrieval-Augmented Large Language Models

Enhancing Large Language Models with Pseudo- and Multisource- Knowledge Graphs for Open-ended Question Answering

Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

Language Models-enhanced Semantic Topology Representation Learning for Temporal Knowledge Graph Extrapolation

Retrieval and Reasoning on KGs: Integrate Knowledge Graphs into Large Language Models for Complex Question Answering

Think and Retrieval: A Hypothesis Knowledge Graph Enhanced Medical Large Language Models

Augmented Large Language Models with Parametric Knowledge Guiding

Self-Knowledge Guided Retrieval Augmentation for Large Language Models

Clue-Guided Path Exploration: Optimizing Knowledge Graph Retrieval with Large Language Models to Address the Information Black Box Challenge

KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation

Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation

Retrieve Anything To Augment Large Language Models

CogMG: Collaborative Augmentation Between Large Language Model and Knowledge Graph

Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering

Large Language Models Can Better Understand Knowledge Graphs Than We Thought