AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

Anselm Paulus,Arman Zharmagambetov,Chuan Guo,Brandon Amos,Yuandong Tian

2024-04-22

Abstract:While recently Large Language Models (LLMs) have achieved remarkable successes, they are vulnerable to certain jailbreaking attacks that lead to generation of inappropriate or harmful content. Manual red-teaming requires finding adversarial prompts that cause such jailbreaking, e.g. by appending a suffix to a given instruction, which is inefficient and time-consuming. On the other hand, automatic adversarial prompt generation often leads to semantically meaningless attacks that can easily be detected by perplexity-based filters, may require gradient information from the TargetLLM, or do not scale well due to time-consuming discrete optimization processes over the token space. In this paper, we present a novel method that uses another LLM, called the AdvPrompter, to generate human-readable adversarial prompts in seconds, $\sim800\times$ faster than existing optimization-based approaches. We train the AdvPrompter using a novel algorithm that does not require access to the gradients of the TargetLLM. This process alternates between two steps: (1) generating high-quality target adversarial suffixes by optimizing the AdvPrompter predictions, and (2) low-rank fine-tuning of the AdvPrompter with the generated adversarial suffixes. The trained AdvPrompter generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response. Experimental results on popular open source TargetLLMs show state-of-the-art results on the AdvBench dataset, that also transfer to closed-source black-box LLM APIs. Further, we demonstrate that by fine-tuning on a synthetic dataset generated by AdvPrompter, LLMs can be made more robust against jailbreaking attacks while maintaining performance, i.e. high MMLU scores.

Cryptography and Security,Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

This paper proposes a new method called AdvPrompter for quickly generating adversarial prompts to test the security of large language models (LLMs). LLMs have shown impressive performance in various tasks but there is a risk of them being maliciously used to generate inappropriate or harmful content. Existing manual and automated methods are inefficient or easily detectable in detecting these attacks, known as "jailbreaking attacks."

AdvPrompter, trained using another LLM, is able to generate human-readable adversarial prompts in a matter of seconds, which is approximately 800 times faster than existing optimization-based methods. The training process involves two steps: optimizing AdvPrompter's predictions to generate high-quality target adversarial suffixes, and fine-tuning AdvPrompter using these suffixes with low rank. The generated suffixes induced the target LLM to produce harmful responses without altering the intended meaning of the input prompt.

Experiments show that AdvPrompter achieves state-of-the-art results on the AdvBench dataset and also demonstrates transferability to closed-source black-box LLM APIs. Furthermore, fine-tuning the LLM on a synthetic dataset generated by AdvPrompter improves its robustness against jailbreaking attacks while maintaining performance.

In summary, this paper addresses the problem of efficiently and quickly generating adversarial prompts to identify and prevent potential security vulnerabilities in large language models.