Abstract:While recently Large Language Models (LLMs) have achieved remarkable successes, they are vulnerable to certain jailbreaking attacks that lead to generation of inappropriate or harmful content. Manual red-teaming requires finding adversarial prompts that cause such jailbreaking, e.g. by appending a suffix to a given instruction, which is inefficient and time-consuming. On the other hand, automatic adversarial prompt generation often leads to semantically meaningless attacks that can easily be detected by perplexity-based filters, may require gradient information from the TargetLLM, or do not scale well due to time-consuming discrete optimization processes over the token space. In this paper, we present a novel method that uses another LLM, called the AdvPrompter, to generate human-readable adversarial prompts in seconds, $\sim800\times$ faster than existing optimization-based approaches. We train the AdvPrompter using a novel algorithm that does not require access to the gradients of the TargetLLM. This process alternates between two steps: (1) generating high-quality target adversarial suffixes by optimizing the AdvPrompter predictions, and (2) low-rank fine-tuning of the AdvPrompter with the generated adversarial suffixes. The trained AdvPrompter generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response. Experimental results on popular open source TargetLLMs show state-of-the-art results on the AdvBench dataset, that also transfer to closed-source black-box LLM APIs. Further, we demonstrate that by fine-tuning on a synthetic dataset generated by AdvPrompter, LLMs can be made more robust against jailbreaking attacks while maintaining performance, i.e. high MMLU scores.

A Prompt-Based Approach to Adversarial Example Generation and Robustness Enhancement

A Prompting-based Approach for Adversarial Example Generation and Robustness Enhancement

Prompt Makes mask Language Models Better Adversarial Attackers

Generating Prompt-Based Adversarial Text Examples Via Variable Neighborhood Search

$\textit{LinkPrompt}$: Natural and Universal Adversarial Attacks on Prompt-based Language Models

PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

Robustness-aware Automatic Prompt Optimization

PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models

Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models

PoisonPrompt: Backdoor Attack on Prompt-based Large Language Models

UTPrompt: Cross-Task Backdoor Prompt Attacks Based on Universal Triggers

Exploring the Universal Vulnerability of Prompt-based Learning Paradigm

Query-Based Adversarial Prompt Generation

A Novel Backdoor Scenario Target the Vulnerability of Prompt-as-a-Service for Code Intelligence Models

Model-tuning Via Prompts Makes NLP Models Adversarially Robust

PromptFix: Few-shot Backdoor Removal via Adversarial Prompt Tuning

PRSA: PRompt Stealing Attacks against Large Language Models

Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information

Prompt Stealing Attacks Against Large Language Models

AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs