Abstract:Large language models (LLMs) have shown great success in various Natural Language Processing (NLP) tasks, whist they still need updates after deployment to fix errors or keep pace with the changing knowledge in the world. Researchers formulate such problem as Model Editing and have developed various editors focusing on different axes of editing properties. However, current editors can hardly support all properties and rely on heavy computational resources. In this paper, we propose a plug-in Model Editing method based on neuron-indexed dynamic LoRA (MELO), which alters the behavior of language models by dynamically activating certain LoRA blocks according to the index built in an inner vector database. Our method satisfies various editing properties with high efficiency and can be easily integrated into multiple LLM backbones. Experimental results show that our proposed MELO achieves state-of-the-art editing performance on three sequential editing tasks (document classification, question answering and hallucination correction), while requires the least trainable parameters and computational cost.

What problem does this paper attempt to address?

The paper attempts to address the issue that large language models (LLMs) need to be updated after deployment to correct errors or adapt to the ever-changing world knowledge. However, frequently retraining or fine-tuning these models becomes impractical due to the high computational costs. Therefore, researchers have proposed the problem of model editing and developed various editors to meet different editing attribute requirements. However, existing editors find it difficult to support all editing attributes and rely on a large amount of computational resources. To address these issues, the paper proposes a plugin-based model editing method called Neuron Indexing Dynamic LoRA (MELO). This method dynamically activates specific LoRA blocks by indexing in an internal vector database, thereby changing the behavior of the language model. The MELO method is highly efficient, can be easily integrated into the backbone of multiple large language models (LLMs), and achieves state-of-the-art editing performance on three consecutive editing tasks (document classification, question answering, and hallucination correction) while requiring the least trainable parameters and computational cost. Specifically, MELO aims to achieve the following editing attributes: 1. **Editing Success**: Each batch of edits is trained with a set of unique LoRA blocks, which are accurately invoked during inference to handle inputs within the range. 2. **Locality**: By constructing an internal vector database to identify the editing range, it ensures that inputs outside the range retain the original predictions. 3. **Generality**: Different radii of semantic clusters are constructed to cover related edits. Once an input falls within the range of a cluster, the corresponding LoRA block is activated. 4. **Sequential Editing**: Different batches of edits are trained using non-overlapping LoRA blocks, addressing the issue of catastrophic forgetting of previous edits. 5. **Efficiency**: MELO uses only small dynamic LoRA blocks of partial rank for editing, allowing a large number of edits to be learned with very few parameters. Through experimental validation, MELO performs excellently on tasks such as document classification, question answering, and hallucination correction. It not only outperforms existing methods in editing performance but also has significant advantages in computational efficiency and parameter usage.

MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA

MEMLA: Enhancing Multilingual Knowledge Editing with Neuron-Masked Low-Rank Adaptation

Robust and Scalable Model Editing for Large Language Models

Neuron-Level Sequential Editing for Large Language Models

On the Robustness of Editing Large Language Models

Language Anisotropic Cross-Lingual Model Editing

Is it Possible to Edit Large Language Models Robustly?

AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models

Editing Large Language Models: Problems, Methods, and Opportunities

Multilingual Knowledge Editing with Language-Agnostic Factual Neurons

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models

O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing

Model Editing for LLMs4Code: How Far are We?

Massive Editing for Large Language Models via Meta Learning

Perturbation-Restrained Sequential Model Editing

Model Editing Can Hurt General Abilities of Large Language Models

Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models

MPN: Leveraging Multilingual Patch Neuron for Cross-lingual Model Editing

MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors

Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generation

Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue