Abstract:Large Language Models (LLMs) have shown proficiency in question-answering tasks but often struggle to integrate real-time knowledge, leading to potentially outdated or inaccurate responses. This problem becomes even more challenging when dealing with multi-hop questions, since they require LLMs to update and integrate multiple knowledge pieces relevant to the questions. To tackle the problem, we propose the Retrieval-Augmented model Editing (RAE) framework for multi-hop question answering. RAE first retrieves edited facts and then refines the language model through in-context learning. Specifically, our retrieval approach, based on mutual information maximization, leverages the reasoning abilities of LLMs to identify chain facts that traditional similarity-based searches might miss. In addition, our framework includes a pruning strategy to eliminate redundant information from the retrieved facts, which enhances the editing accuracy and mitigates the hallucination problem. Our framework is supported by theoretical justification for its fact retrieval efficacy. Finally, comprehensive evaluation across various LLMs validates RAE's ability in providing accurate answers with updated knowledge. Our code is available at: <a class="link-external link-https" href="https://github.com/sycny/RAE" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

This paper attempts to address the issue of integrating real-time knowledge in multi-hop question answering tasks faced by large language models (LLMs). Specifically, while LLMs perform well in question answering tasks, they often generate outdated or inaccurate answers when dealing with multi-hop questions that require updating and integrating multiple relevant pieces of knowledge. To tackle this challenge, the authors propose a framework called "Retrieval-Augmented model Editing" (RAE). ### Main Issues: 1. **Real-time Knowledge Integration**: LLMs rely on pre-trained knowledge, which may lead to outdated or inaccurate answers. 2. **Multi-hop Question Handling**: Multi-hop questions require the model to update and integrate multiple relevant pieces of knowledge, which is difficult to achieve with existing methods. 3. **Retrieval Complexity**: Traditional similarity-based retrieval methods often fail to retrieve relevant facts when dealing with complex multi-hop questions. ### Solutions: - **RAE Framework**: By using a retrieval-augmented approach, it first retrieves edited facts and then fine-tunes the language model through contextual learning. - **Mutual Information Maximization**: Utilizes a mutual information maximization strategy to retrieve a chain of facts related to the question, ensuring that the retrieved facts are highly relevant to the question. - **Redundant Information Pruning**: Identifies and removes redundant information through the uncertainty of the model's output, improving the accuracy of the edits and reducing hallucination issues. ### Specific Methods: 1. **Knowledge Graph Construction**: Stores the edited facts in a knowledge graph, using the connections between entities to narrow down the retrieval scope. 2. **Next Token Prediction**: Utilizes the next token prediction capability of LLMs to predict the next fact in the fact chain. 3. **Mutual Information Metric**: Uses mutual information as a retrieval metric to ensure that the retrieved facts are relevant to the question. 4. **Uncertainty Pruning**: Identifies and removes redundant information by calculating the entropy of the model's output, improving the accuracy of the edits. ### Experimental Validation: - **Dataset**: Experiments were conducted using the MQUAKE-CF dataset, which contains multi-hop questions based on real-world facts. - **Experimental Results**: The experiments validated the effectiveness and accuracy of the RAE framework in multi-hop question answering tasks, providing updated and accurate answers. In summary, this paper proposes a novel approach that addresses the issue of integrating real-time knowledge in multi-hop question answering tasks faced by LLMs through retrieval-augmented and knowledge graph techniques.

Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering

Knowledge Editing with Dynamic Knowledge Graphs for Multi-hop Question Answering

Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing

Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation

Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding

MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions

Retrieval-augmented Multilingual Knowledge Editing

LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

Hierarchical Retrieval-Augmented Generation Model with Rethink for Multi-hop Question Answering

Enhancing Large Language Models with Pseudo- and Multisource- Knowledge Graphs for Open-ended Question Answering

Layered Query Retrieval: an Adaptive Framework for Retrieval-Augmented Generation in Complex Question Answering for Large Language Models

Review-Then-Refine: A Dynamic Framework for Multi-Hop Question Answering with Temporal Adaptability

EffiQA: Efficient Question-Answering with Strategic Multi-Model Collaboration on Knowledge Graphs

Leveraging Logical Rules in Knowledge Editing: A Cherry on the Top

Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

Efficient Question Answering Based on Language Models and Knowledge Graphs

Robust and Scalable Model Editing for Large Language Models

CoQ:AN Empirical Framework for Multi-hop Question Answering Empowered by Large Language Models

MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language

Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs