Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs

Junjie Wang,Mingyang Chen,Binbin Hu,Dan Yang,Ziqi Liu,Yue Shen,Peng Wei,Zhiqiang Zhang,Jinjie Gu,Jun Zhou,Jeff Z. Pan,Wen Zhang,Huajun Chen

2024-10-23

Abstract:Improving the performance of large language models (LLMs) in complex question-answering (QA) scenarios has always been a research focal point. Recent studies have attempted to enhance LLMs' performance by combining step-wise planning with external retrieval. While effective for advanced models like GPT-3.5, smaller LLMs face challenges in decomposing complex questions, necessitating supervised fine-tuning. Previous work has relied on manual annotation and knowledge distillation from teacher LLMs, which are time-consuming and not accurate enough. In this paper, we introduce a novel framework for enhancing LLMs' planning capabilities by using planning data derived from knowledge graphs (KGs). LLMs fine-tuned with this data have improved planning capabilities, better equipping them to handle complex QA tasks that involve retrieval. Evaluations on multiple datasets, including our newly proposed benchmark, highlight the effectiveness of our framework and the benefits of KG-derived planning data.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem this paper attempts to address is: how to improve the performance of large language models (LLMs) in complex question-answering tasks, especially for smaller LLMs that face challenges in decomposing complex questions. Existing methods typically rely on manual annotation or distilling knowledge from teacher models, which are both time-consuming and inaccurate. Therefore, this paper proposes a new framework that enhances the planning capabilities of LLMs by using planning data generated from knowledge graphs (KGs), thereby improving their performance in complex question-answering tasks.

Specifically, the main contributions of the paper include:
1. **Introducing the LPKG framework**: This framework leverages planning data extracted from knowledge graphs to enhance the planning capabilities of LLMs.
2. **Constructing the CLQA-Wiki benchmark dataset**: This is a comprehensive and challenging complex logical question-answering benchmark dataset that covers multi-hop, comparison, intersection, and union types of questions.
3. **Experimental validation**: On multiple traditional complex question-answering benchmark datasets, the LPKG framework outperforms existing baseline methods and significantly improves the planning capabilities of LLMs by using KG-sourced planning data.

Through these contributions, the paper aims to provide a more efficient and accurate method to enhance the performance of LLMs in complex question-answering tasks.