Abstract:Large Language Models (LLMs) demonstrate remarkable capabilities, yet struggle with hallucination and outdated knowledge when tasked with complex knowledge reasoning, resulting in factually incorrect outputs. Previous studies have attempted to mitigate it by retrieving factual knowledge from large-scale knowledge graphs (KGs) to assist LLMs in logical reasoning and prediction of answers. However, this kind of approach often introduces noise and irrelevant data, especially in situations with extensive context from multiple knowledge aspects. In this way, LLM attention can be potentially mislead from question and relevant information. In our study, we introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework. This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings. The Amar framework comprises two key sub-components: 1) a self-alignment module that aligns commonalities among entities, relations, and subgraphs to enhance retrieved text, thereby reducing noise interference; 2) a relevance gating module that employs a soft gate to learn the relevance score between question and multi-aspect retrieved data, to determine which information should be used to enhance LLMs' output, or even filtered altogether. Our method has achieved state-of-the-art performance on two common datasets, WebQSP and CWQ, showing a 1.9\% improvement in accuracy over its best competitor and a 6.6\% improvement in logical form generation over a method that directly uses retrieved text as context prompts. These results demonstrate the effectiveness of Amar in improving the reasoning of LLMs.

Knowledge Graph-augmented Language Models for Complex Question Answering

Retrieval and Reasoning on KGs: Integrate Knowledge Graphs into Large Language Models for Complex Question Answering

Enhancing Large Language Models with Knowledge Graphs for Robust Question Answering

A Method for Complex Question-Answering over Knowledge Graph

Multi-hop Question Answering over Knowledge Graphs using Large Language Models

ReLMKG: Reasoning with Pre-Trained Language Models and Knowledge Graphs for Complex Question Answering

Enhancing Large Language Models with Pseudo- and Multisource- Knowledge Graphs for Open-ended Question Answering

A Framework of Knowledge Graph-Enhanced Large Language Model Based on Question Decomposition and Atomic Retrieval

Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study over Open-ended Question Answering

Efficient Question Answering Based on Language Models and Knowledge Graphs

Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering

MST5 -- Multilingual Question Answering over Knowledge Graphs

Knowledge Graph Question Answering with semantic oriented fusion model.

QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering

Application of large language models based on knowledge graphs in question-answering systems: A review

Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs

KG-CoT: Chain-of-Thought Prompting of Large Language Models over Knowledge Graphs for Knowledge-Aware Question Answering

Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts