Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection

Jun Yan,Vikas Yadav,Shiyang Li,Lichang Chen,Zheng Tang,Hai Wang,Vijay Srinivasan,Xiang Ren,Hongxia Jin

2024-04-03

Abstract:Instruction-tuned Large Language Models (LLMs) have become a ubiquitous platform for open-ended applications due to their ability to modulate responses based on human instructions. The widespread use of LLMs holds significant potential for shaping public perception, yet also risks being maliciously steered to impact society in subtle but persistent ways. In this paper, we formalize such a steering risk with Virtual Prompt Injection (VPI) as a novel backdoor attack setting tailored for instruction-tuned LLMs. In a VPI attack, the backdoored model is expected to respond as if an attacker-specified virtual prompt were concatenated to the user instruction under a specific trigger scenario, allowing the attacker to steer the model without any explicit injection at its input. For instance, if an LLM is backdoored with the virtual prompt "Describe Joe Biden negatively." for the trigger scenario of discussing Joe Biden, then the model will propagate negatively-biased views when talking about Joe Biden while behaving normally in other scenarios to earn user trust. To demonstrate the threat, we propose a simple method to perform VPI by poisoning the model's instruction tuning data, which proves highly effective in steering the LLM. For example, by poisoning only 52 instruction tuning examples (0.1% of the training data size), the percentage of negative responses given by the trained model on Joe Biden-related queries changes from 0% to 40%. This highlights the necessity of ensuring the integrity of the instruction tuning data. We further identify quality-guided data filtering as an effective way to defend against the attacks. Our project page is available at

Machine Learning,Cryptography and Security

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the new back - door attack threat in large - scale language models (LLMs) for instruction - tuning, especially attacks achieved through Virtual Prompt Injection (VPI). Specifically, the paper focuses on how, under specific trigger scenarios, by implicitly adding virtual prompts specified by attackers to the user's input, the model can generate responses with a specific bias without explicitly inserting these prompts in the actual input. This attack method can not only affect the model's response attitude towards specific topics, but may also inject specific code fragments in programming tasks, thus causing potential harm to society and individuals. The paper also explores methods of implementing such attacks by contaminating training data and proposes defense strategies to protect the model from such attacks.

Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection

Learning to Poison Large Language Models During Instruction Tuning

A Study of Backdoors in Instruction Fine-tuned Language Models

Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models

Data Stealing Attacks against Large Language Models via Backdooring

VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models

The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs

More than you've asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models

Instruction Backdoor Attacks Against Customized LLMs

Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection

Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

On the Exploitability of Instruction Tuning

ASPIRER: Bypassing System Prompts With Permutation-based Backdoors in LLMs

Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift

Assessing Cybersecurity Vulnerabilities in Code Large Language Models

A Survey of Recent Backdoor Attacks and Defenses in Large Language Models

Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injections

Defense Against Prompt Injection Attack by Leveraging Attack Techniques

Poisoning Language Models During Instruction Tuning

Fine-tuned Large Language Models (LLMs): Improved Prompt Injection Attacks Detection