Abstract:The general capabilities of large language models (LLMs) make them the infrastructure for various AI applications, but updating their inner knowledge requires significant resources. Recent model editing is a promising technique for efficiently updating a small amount of knowledge of LLMs and has attracted much attention. In particular, local editing methods, which directly update model parameters, are more suitable for updating a small amount of knowledge. Local editing methods update weights by computing least squares closed-form solutions and identify edited knowledge by vector-level matching in inference, which achieve promising results. However, these methods still require a lot of time and resources to complete the computation. Moreover, vector-level matching lacks reliability, and such updates disrupt the original organization of the model's parameters. To address these issues, we propose an detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching and adds them to the subject word embeddings in Transformer input. To get these editing embeddings, we propose optimizing then suppressing fusion method, which first optimizes learnable embedding vectors for the editing target and then suppresses the Knowledge Embedding Dimensions (KEDs) to obtain final editing embeddings. We thus propose SWEA$\oplus$OS method for editing factual knowledge in LLMs. We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$\oplus$OS on the \textsc{CounterFact} and zsRE datasets. To further validate the reasoning ability of SWEA$\oplus$OS in editing knowledge, we evaluate it on the more complex \textsc{RippleEdits} benchmark. The results demonstrate that SWEA$\oplus$OS possesses SOTA reasoning ability.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to update the factual knowledge in large - language models (LLMs) efficiently and reliably while avoiding unnecessary interference with the original parameters of the model. Specifically, the current methods have the following problems when updating a small amount of knowledge: 1. **Low efficiency**: The existing local editing methods require a large amount of time and resources to calculate all vectors to solve the least - squares problem. 2. **Insufficient reliability**: These methods use vector - level matching to identify the edited knowledge, but this method is not reliable enough and it is easy to mis - identify the unedited knowledge as the edited knowledge. 3. **Lack of protection**: Directly modifying the model weights may destroy the original organizational structure of the model parameters and affect the generalization ability of the model. To address these problems, the author proposes a detachable and extensible subject - word - embedding - adjustment framework (SWEA) and obtains the edited embedding through the optimize - then - suppress (OS) fusion method. The SWEA ⊕OS method solves the above problems in the following ways: - **Improve efficiency**: Only the edited embedding needs to be calculated, reducing the amount of calculation. - **Enhance reliability**: Use a token - level matching method, which is more sensitive than vector - level matching and can more accurately identify editing instances. - **Protect the original parameters**: Instead of directly modifying the model weights, the knowledge update is achieved by adjusting the input word embeddings, thereby protecting the original parameters and generalization ability of the model. ### Specific implementation methods 1. **Optimize - then - suppress (OS) fusion method**: - Optimize the learnable embedding vectors to achieve the editing goal. - Suppress the knowledge - embedding - dimensions (KEDs) specific to the subject to reduce their impact on the expression of new knowledge. 2. **Subject - word - embedding - adjustment framework (SWEA)**: - In the inference stage, add the edited embedding to the input subject - word - embedding to obtain the final input embedding. - Use a token - level matching method to ensure the accuracy of the editing. ### Experimental results The experimental results show that SWEA ⊕OS exhibits superior performance in the COUNTERFACT and zsRE datasets as well as the RippleEdits benchmark test, especially being particularly stable in batch - editing and sequential - batch - editing tasks. Although SWEA ⊕OS has a certain impact on the generation ability of the model, it is significantly superior to the existing methods in terms of editing effect, generalization ability and specificity. ### Summary This paper proposes a new model - editing method SWEA ⊕OS, aiming to update the factual knowledge in LLMs efficiently and reliably while protecting the original parameters and generalization ability of the model. The experimental results show that SWEA ⊕OS performs well in multiple datasets and benchmark tests and has broad application prospects.

SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering

WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models

Uncovering Overfitting in Large Language Model Editing

StruEdit: Structured Outputs Enable the Fast and Accurate Knowledge Editing for Large Language Models

A Comprehensive Study of Knowledge Editing for Large Language Models

Eva-KELLM: A New Benchmark for Evaluating Knowledge Editing of LLMs

Cross-Lingual Knowledge Editing in Large Language Models

Multilingual Knowledge Editing with Language-Agnostic Factual Neurons

Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing

Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering

FAME: Towards Factual Multi-Task Model Editing

Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models

Editing Language Model-based Knowledge Graph Embeddings

EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models

Robust and Scalable Model Editing for Large Language Models

Commonsense Knowledge Editing Based on Free-Text in LLMs

Editing Conceptual Knowledge for Large Language Models

AKEW: Assessing Knowledge Editing in the Wild

Cross-Lingual Multi-Hop Knowledge Editing -- Benchmarks, Analysis and a Simple Contrastive Learning based Approach