Abstract:In recent times, large language models (LLMs) have showcased remarkable capabilities. However, updating their knowledge poses challenges, potentially leading to inaccuracies when confronted with unfamiliar queries. To address this issue, integrating external knowledge bases such as knowledge graphs with large language models is a viable approach. The key challenge lies in extracting the required knowledge from knowledge graphs based on natural language, demanding high semantic understanding. Therefore, researchers are considering leveraging large language models directly for knowledge retrieval from these graphs. Current efforts typically rely on the comprehensive problem-solving capabilities of large language models. We argue that a problem we term the 'information black box' can significantly impact the practical effectiveness of such methods. Moreover, this kind of methods is less effective for scenarios where the questions are unfamiliar to the large language models. In this paper, we propose a Clue-Guided Path Exploration (CGPE) framework to optimize knowledge retrieval based on large language models. By addressing the 'information black box' issue and employing single-task approaches instead of complex tasks, we have enhanced the accuracy and efficiency of using large language models for retrieving knowledge graphs. Experiments on open-source datasets reveal that CGPE outperforms previous methods and is highly applicable to LLMs with fewer parameters. In some instances, even ChatGLM3, with its 6 billion parameters, can rival the performance of GPT-4. Furthermore, the results indicate a minimal invocation frequency of CGPE on LLMs, suggesting reduced computational overhead. For organizations and individuals facing constraints in computational resources, our research offers significant practical value.

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper aims to address the information black box problem faced by large language models (LLMs) in knowledge graph retrieval. Specifically, current methods often rely on the comprehensive problem-solving capabilities of LLMs but perform poorly when dealing with unfamiliar issues. Additionally, these methods typically require substantial computational resources, limiting their widespread application. ### Main Contributions 1. **Identifying the Information Black Box Problem**: The paper is the first to point out the "information black box" problem in existing methods, where it is unclear what key information LLMs extract or use during the information analysis and processing stages when exploring knowledge graphs. This can lead to information omission or duplication, affecting the efficiency of knowledge graph exploration. 2. **Proposing the CGPE Framework**: The paper proposes a Clue-Guided Path Exploration (CGPE) framework, which explicitly extracts necessary information from the problem and uses it as clues to guide the entire path exploration process. This method reduces reliance on the comprehensive capabilities of LLMs, improving the accuracy and efficiency of knowledge graph exploration. 3. **Single Task Information Matching**: CGPE uses a single "information matching" task to complete each step of the path exploration, rather than complex multi-tasking. This approach achieves good results even with LLMs with fewer parameters, enhancing its practical application value. 4. **Experimental Validation**: Experimental results show that CGPE outperforms existing methods on multiple datasets, including higher accuracy, lower LLM invocation frequency, and effective performance on LLMs with fewer parameters. These advantages highlight the practical application value of CGPE. ### Solution 1. **Explicit Extraction of Necessary Information**: CGPE explicitly extracts necessary entities and relationships from the problem and uses this information as clues to guide subsequent path exploration. 2. **Path Exploration Process**: By iteratively matching unused clues, CGPE explores each subsequent node step by step until a complete knowledge path is found. This method ensures the systematic and efficient exploration of paths. 3. **Information Matching Task**: CGPE uses a single "information matching" task to complete each step of the path exploration, reducing reliance on the comprehensive capabilities of LLMs. This method is not only suitable for unfamiliar problems but also significantly reduces the demand for computational resources. ### Experimental Results - **MOOC Q&A Dataset**: On unfamiliar educational domain questions, CGPE significantly outperforms baseline methods and the state-of-the-art method ToG, especially on true/false and query questions. - **WebQuestions and WebQSP Datasets**: Even on relatively familiar questions, CGPE still performs excellently, particularly outperforming ToG on partial match and exact match metrics. - **Computational Cost**: CGPE significantly reduces the frequency of LLM invocations during path exploration compared to ToG, lowering the consumption of computational resources. ### Conclusion The CGPE framework proposed in the paper addresses the information black box problem, improving the accuracy and efficiency of LLMs in knowledge graph retrieval, especially in scenarios with limited computational resources. Experimental results validate the effectiveness and practicality of CGPE, providing new directions for future research.

Clue-Guided Path Exploration: Optimizing Knowledge Graph Retrieval with Large Language Models to Address the Information Black Box Challenge

Knowledge Graph-Enhanced Large Language Models via Path Selection

Reasoning on Efficient Knowledge Paths:Knowledge Graph Guides Large Language Model for Domain Question Answering

Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs

KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases

Large Language Models and Knowledge Graphs: Opportunities and Challenges

Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning

Augmented Large Language Models with Parametric Knowledge Guiding

ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling

Large Language Models Can Better Understand Knowledge Graphs Than We Thought

Exploring Large Language Models for Knowledge Graph Completion

Graph Neural Prompting with Large Language Models

KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph

Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs

Combining Knowledge Graphs and Large Language Models

Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation

Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling

Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs

KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion