Abstract:Large language models (LLMs) are pivotal in advancing natural language processing (NLP) tasks, yet their efficacy is hampered by inaccuracies and outdated knowledge. Model editing emerges as a promising solution to address these challenges. However, existing editing methods struggle to track and incorporate changes in knowledge associated with edits, which limits the generalization ability of postedit LLMs in processing edited knowledge. To tackle these problems, we propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME. Specifically, we first utilize a knowledge graph augmentation module to uncover associated knowledge that has changed due to editing, obtaining its internal representations within LLMs. This approach allows knowledge alterations within LLMs to be reflected through an external graph structure. Subsequently, we design a graph-based knowledge edit module to integrate structured knowledge into the model editing. This ensures that the updated parameters reflect not only the modifications of the edited knowledge but also the changes in other associated knowledge resulting from the editing process. Comprehensive experiments conducted on GPT-J and GPT-2 XL demonstrate that GLAME significantly improves the generalization capabilities of post-edit LLMs in employing edited knowledge.

What problem does this paper attempt to address?

The paper primarily addresses the issue of inaccurate or outdated knowledge in large language models (LLMs) and proposes a new model editing method aimed at improving the generalization ability of the model to the edited knowledge. Specifically, the paper addresses the following issues: 1. **Knowledge Accuracy and Update**: Although current LLMs perform well in natural language processing tasks, their internal knowledge may be inaccurate or outdated, limiting the model's capabilities. To address this, researchers have proposed model editing techniques to update this knowledge. 2. **Capturing Changes in Related Knowledge**: Existing model editing methods often overlook the changes in related knowledge triggered by the modification of target knowledge. For example, if "LeBron James plays for the Miami Heat" is edited to "LeBron James plays for the Los Angeles Lakers," other related knowledge, such as "LeBron James works in Miami," should also be updated to "LeBron James works in Los Angeles." 3. **Limitations of Model Editing**: Current editing methods mainly focus on modifying individual pieces of knowledge and fail to effectively capture and utilize the changes in related knowledge caused by the edits. This limits the model's ability to handle these changes after editing. To address the above issues, the paper proposes a new method called **GLAME** (Graph-enhanced Large Language Model Editing). The main contributions of this method include: - **Importance of Related Knowledge Changes**: Emphasizes the importance of capturing changes in related knowledge that occur during the model editing process. - **Combining Knowledge Graphs for Editing**: Introduces knowledge graphs to structure the knowledge changes caused by the edits and integrates them into specific parameters to enhance the editing effect. - **Experimental Validation**: Conducted extensive experiments on GPT-2 XL and GPT-J models to validate the effectiveness of the proposed method. In summary, the paper effectively addresses the limitations of existing model editing techniques in handling changes in related knowledge by proposing a knowledge graph-enhanced model editing method (GLAME), thereby improving the generalization ability of the edited model and the efficiency of utilizing the edited knowledge.

Knowledge Graph Enhanced Large Language Model Editing

Knowledge Graph Enhanced Large Language Model Editing

A Comprehensive Study of Knowledge Editing for Large Language Models

Knowledge Editing for Large Language Models: A Survey

Editing Conceptual Knowledge for Large Language Models

ConceptEdit: Conceptualization-Augmented Knowledge Editing in Large Language Models for Commonsense Reasoning

EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models

Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing

Learning to Edit: Aligning LLMs with Knowledge Editing

Knowledge Editing on Black-box Large Language Models

Cross-Lingual Knowledge Editing in Large Language Models

ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling

Unveiling the Pitfalls of Knowledge Editing for Large Language Models

Editing Language Model-based Knowledge Graph Embeddings

Uncovering Overfitting in Large Language Model Editing