Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models

Rui Ye,Jingyi Chai,Xiangrui Liu,Yaodong Yang,Yanfeng Wang,Siheng Chen

2024-06-15

Abstract:Federated learning (FL) enables multiple parties to collaboratively fine-tune an large language model (LLM) without the need of direct data sharing. Ideally, by training on decentralized data that is aligned with human preferences and safety principles, federated instruction tuning can result in an LLM that could behave in a helpful and safe manner. In this paper, we for the first time reveal the vulnerability of safety alignment in FedIT by proposing a simple, stealthy, yet effective safety attack method. Specifically, the malicious clients could automatically generate attack data without involving manual efforts and attack the FedIT system by training their local LLMs on such attack data. Unfortunately, this proposed safety attack not only can compromise the safety alignment of LLM trained via FedIT, but also can not be effectively defended against by many existing FL defense methods. Targeting this, we further propose a post-hoc defense method, which could rely on a fully automated pipeline: generation of defense data and further fine-tuning of the LLM. Extensive experiments show that our safety attack method can significantly compromise the LLM's safety alignment (e.g., reduce safety rate by 70\%), which can not be effectively defended by existing defense methods (at most 4\% absolute improvement), while our safety defense method can significantly enhance the attacked LLM's safety alignment (at most 69\% absolute improvement).

Computation and Language,Artificial Intelligence,Cryptography and Security,Multiagent Systems

What problem does this paper attempt to address?

This paper is primarily dedicated to revealing potential security issues in Federated Instruction Tuning (FedIT) within large language models (LLMs) and proposing corresponding attack and defense methods. Specifically:

1. **Revealing Vulnerabilities**: The paper is the first to reveal the vulnerability of FedIT in terms of security alignment, proposing a simple, covert, yet effective security attack method. This attack involves malicious clients generating misaligned data to train local LLMs, thereby compromising the security of the global LLM.

2. **Attack Method**: Malicious clients can launch attacks by automatically generating misaligned data without human intervention. Experiments show that this attack method can significantly reduce the security alignment of LLMs, with a maximum security reduction of 70%.

3. **Failure of Existing Defense Methods**: Many existing Federated Learning (FL) defense methods cannot effectively resist this security attack, providing at most a 4% improvement in security.

4. **New Defense Method**: To address the above issues, the paper proposes a post-processing defense method based on an automated data generation process on the server side, further fine-tuning the attacked LLM to enhance its security alignment. Experimental results show that this method can significantly improve the security of attacked LLMs, with an absolute improvement of up to 69%.

In summary, this paper aims to reveal and address the security issues of large language models in a federated learning environment by proposing new attack and defense methods.