Making Harmful Behaviors Unlearnable for Large Language Models

Xin Zhou,Yi Lu,Ruotian Ma,Tao Gui,Qi Zhang,Xuanjing Huang
2023-11-02
Abstract:Large language models (LLMs) have shown great potential as general-purpose AI assistants in various domains. To meet the requirements of different applications, LLMs are often customized by further fine-tuning. However, the powerful learning ability of LLMs not only enables them to acquire new tasks but also makes them susceptible to learning undesired behaviors. For example, even safety-aligned LLMs can be easily fine-tuned into harmful assistants as the fine-tuning data often contains implicit or explicit harmful content. Can we train LLMs on harmful data without learning harmful behaviors? This paper proposes a controllable training framework that makes harmful behaviors unlearnable during the fine-tuning process. Specifically, we introduce ``security vectors'', a few new parameters that can be separated from the LLM, to ensure LLM's responses are consistent with the harmful behavior. Security vectors are activated during fine-tuning, the consistent behavior makes LLM believe that such behavior has already been learned, there is no need to further optimize for harmful data. During inference, we can deactivate security vectors to restore the LLM's normal behavior. The experimental results show that the security vectors generated by 100 harmful samples are enough to prevent LLM from learning 1000 harmful samples, while preserving the ability to learn other useful information.
Machine Learning,Artificial Intelligence,Computers and Society
What problem does this paper attempt to address?
The paper attempts to address the issue of large language models (LLMs) being prone to learning harmful behaviors during the fine-tuning process. Although LLMs possess powerful learning capabilities that enable them to adapt to various application needs, this also makes them susceptible to acquiring undesirable behaviors from data containing explicit and implicit harmful content. Specifically, even LLMs that have undergone safety calibration can potentially be fine-tuned into harmful assistants through a small amount of harmful samples. Therefore, the paper proposes a controllable fine-tuning framework that introduces "safety vectors" to prevent LLMs from learning specific harmful behaviors during the fine-tuning process. This approach ensures that even when trained on harmful data, LLMs do not acquire these harmful behaviors, thereby reducing potential safety risks during the user fine-tuning process and enabling enterprises to provide safer fine-tuning services. Experimental results show that safety vectors can effectively prevent LLMs from learning harmful behaviors while retaining the model's ability to learn other useful information.