Abstract:Large Language Models (LLMs) have demonstrated great potential as generalist assistants, showcasing powerful task understanding and problem-solving capabilities. To deploy LLMs as AI assistants, it is crucial that these models exhibit desirable behavioral traits, such as non-toxicity and resilience against jailbreak attempts. Current methods for detoxification or preventing jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), which requires finetuning billions of parameters through gradient descent with substantial computation cost. Furthermore, models modified through SFT and RLHF may deviate from the pretrained models, potentially leading to a degradation in foundational LLM capabilities. In this paper, we observe that surprisingly, directly editing a small subset of parameters can effectively modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreaking. Specifically, for a behavior that we aim to avoid, we employ a linear classifier, which we term the behavior probe, to classify binary behavior labels within the hidden state space of the LLM. Using this probe, we introduce an algorithm to identify a critical subset of LLM parameters that significantly influence this targeted behavior. Then we directly edit these selected parameters by shifting them towards the behavior probe. Such a direct parameter editing method necessitates only inference-level computational resources. Experiments demonstrate that in the representative detoxification task, our approach achieves reductions of up to 90.0\% in toxicity on the RealToxicityPrompts dataset and 49.2\% on ToxiGen, while maintaining the LLM's general capabilities in areas such as common sense, question answering, and mathematics. Our code is available at <a class="link-external link-https" href="https://github.com/lucywang720/model-surgery" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

This paper presents a method called "Model Surgery" for adjusting the behavior of large-scale language models (LLMs) by simply editing a small subset of their parameters, such as eliminating harmful content and preventing attempts to escape, without the need for extensive gradient descent training. Existing methods such as supervised fine-tuning (SFT) and human-feedback-based reinforcement learning (RLHF) are effective but require substantial computational resources and may lead to degradation of the underlying LLM capabilities. The paper observes that adjusting the behavior of LLMs can be effectively achieved by directly editing a small subset of parameters, for example, by training a linear classifier (behavior probe) to identify and categorize behavior labels in the hidden state space. Using this probe, they propose an algorithm to identify the key subset of parameters that influence the target behavior and then directly modify these parameters towards the direction of the behavior probe. This approach only requires inference-level computational resources. Experimental results demonstrate that their method reduces toxicity by 90.0% on the RealToxicityPrompts dataset and by 49.2% on the ToxiGen dataset for representative harmful content elimination tasks, while maintaining the LLM's general capabilities in areas like commonsense reasoning, question answering, and mathematics. The paper also discusses the advantages of Model Surgery compared to other methods such as SFT and RLHF, including reduced computational resource requirements and minor impact on the model's underlying capabilities. Overall, this paper aims to address how to adjust and improve the behavior of models, particularly in eliminating harmful content and enhancing security, by directly editing parameters without compromising the basic performance of LLMs.

Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing

Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models

Detoxifying Large Language Models via Knowledge Editing

Multi-round jailbreak attack on large language models

Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing

Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomes

Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models

Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinement

Making Harmful Behaviors Unlearnable for Large Language Models

Model-Editing-Based Jailbreak against Safety-aligned Large Language Models

Large Language Models can be Strong Self-Detoxifiers

LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B

Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teaming

Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue

Locking Down the Finetuned LLMs Safety

Precision Knowledge Editing: Enhancing Safety in Large Language Models

Model Editing Can Hurt General Abilities of Large Language Models

Legilimens: Practical and Unified Content Moderation for Large Language Model Services

Mitigating Exaggerated Safety in Large Language Models

Systematic Rectification of Language Models via Dead-end Analysis

Rapid Optimization for Jailbreaking LLMs via Subconscious Exploitation and Echopraxia