EffiQA: Efficient Question-Answering with Strategic Multi-Model Collaboration on Knowledge Graphs

Zixuan Dong,Baoyun Peng,Yufei Wang,Jia Fu,Xiaodong Wang,Yongxue Shan,Xin Zhou

2024-09-24

Abstract:While large language models (LLMs) have shown remarkable capabilities in natural language processing, they struggle with complex, multi-step reasoning tasks involving knowledge graphs (KGs). Existing approaches that integrate LLMs and KGs either underutilize the reasoning abilities of LLMs or suffer from prohibitive computational costs due to tight coupling. To address these limitations, we propose a novel collaborative framework named EffiQA that can strike a balance between performance and efficiency via an iterative paradigm. EffiQA consists of three stages: global planning, efficient KG exploration, and self-reflection. Specifically, EffiQA leverages the commonsense capability of LLMs to explore potential reasoning pathways through global planning. Then, it offloads semantic pruning to a small plug-in model for efficient KG exploration. Finally, the exploration results are fed to LLMs for self-reflection to further improve the global planning and efficient KG exploration. Empirical evidence on multiple KBQA benchmarks shows EffiQA's effectiveness, achieving an optimal balance between reasoning accuracy and computational costs. We hope the proposed new framework will pave the way for efficient, knowledge-intensive querying by redefining the integration of LLMs and KGs, fostering future research on knowledge-based question answering.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the challenges faced by large language models (LLMs) when handling complex, multi-step reasoning tasks involving knowledge graphs (KGs). Specifically, existing methods that combine LLMs with KGs either fail to fully utilize the reasoning capabilities of LLMs or incur excessively high computational costs due to tight coupling. To address these issues, the authors propose a new collaborative framework called EffiQA.

EffiQA balances performance and efficiency through three stages:
1. **Global Planning**: Utilizes the common-sense capabilities of LLMs to explore potential reasoning paths and generate exploration instructions.
2. **Efficient KG Exploration**: Employs a small plugin model for semantic pruning to enhance the efficiency of the KG search process.
3. **Self-Reflection**: LLMs reflect on the exploration results to improve global planning and efficient KG exploration.

Experimental results show that EffiQA performs excellently in multiple benchmarks, achieving the best balance between reasoning accuracy and computational cost. This new framework is expected to redefine the standards for LLMs and KGs integration, promoting the development of future knowledge-based question answering research.