Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models

Zhiyuan Yu,Xiaogeng Liu,Shunning Liang,Zach Cameron,Chaowei Xiao,Ning Zhang

2024-10-01

Abstract:Recent advancements in generative AI have enabled ubiquitous access to large language models (LLMs). Empowered by their exceptional capabilities to understand and generate human-like text, these models are being increasingly integrated into our society. At the same time, there are also concerns on the potential misuse of this powerful technology, prompting defensive measures from service providers. To overcome such protection, jailbreaking prompts have recently emerged as one of the most effective mechanisms to circumvent security restrictions and elicit harmful content originally designed to be prohibited. Due to the rapid development of LLMs and their ease of access via natural languages, the frontline of jailbreak prompts is largely seen in online forums and among hobbyists. To gain a better understanding of the threat landscape of semantically meaningful jailbreak prompts, we systemized existing prompts and measured their jailbreak effectiveness empirically. Further, we conducted a user study involving 92 participants with diverse backgrounds to unveil the process of manually creating jailbreak prompts. We observed that users often succeeded in jailbreak prompts generation regardless of their expertise in LLMs. Building on the insights from the user study, we also developed a system using AI as the assistant to automate the process of jailbreak prompt generation.

Cryptography and Security,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the potential security threats brought by the widespread application of large - language models (LLMs) in society, especially by circumventing the security limitations of these models through so - called "jailbreak prompts". Specifically, the paper focuses on the following aspects:

1. **Existing jailbreak prompt strategies and their effectiveness**: The paper systematically collected 448 actually - used jailbreak prompts and extracted 161 malicious queries from them, which deliberately violate OpenAI's policies. Through a thematic analysis of these prompts, they are classified into five categories, each containing unique patterns. In addition, the paper proposes two new evaluation metrics to measure the ability of jailbreak prompts to bypass the LLM security limitations and the level of detail in eliciting harmful responses.

2. **How humans develop and execute semantically meaningful jailbreak attacks**: Through a user study involving 92 participants, the paper explores how people from different backgrounds generate jailbreak prompts. The study found that even inexperienced participants were able to successfully create jailbreak prompts, indicating that there is great potential in leveraging human creativity to manipulate language models in conversations.

3. **Can humans and AI collaborate to automatically generate semantically meaningful jailbreak prompts?**: Based on the observations in the user study, the paper further explores the feasibility of using AI agents to automatically generate jailbreak prompts. Through an interactive framework, the AI agent iteratively applies prompt mutations and tests their impact on the jailbreak effect. Preliminary results show that the framework was able to successfully transform 729 out of 766 previously failed prompts, enabling them to elicit harmful content.

### Main contributions of the paper:
- **Dataset collection and analysis**: The paper collected and analyzed a comprehensive dataset, including 448 actually - used jailbreak prompts and 161 malicious queries, and systematized five categories and ten unique jailbreak patterns through a structured inductive thematic coding process.
- **Effectiveness evaluation of jailbreak prompts**: The paper evaluated the effectiveness of jailbreak prompts on three state - of - the - art commercial models (GPT - 3.5, GPT - 4, and PaLM - 2), using human - annotated output data, and proposed two new statistical metrics to evaluate the jailbreak effect. The analysis found two of the most effective strategies and the existence of universal jailbreak prompts.
- **User study**: Through a user study involving 92 participants, the paper revealed how humans develop and execute semantically meaningful jailbreak attacks, and identified previously undetected jailbreak patterns and methods.
- **Feasibility of automatically generating jailbreak prompts**: The paper developed an interactive framework that automatically optimizes prompts by evaluating the output of the target LLM. Preliminary experiments showed that the framework successfully transformed 729 out of 766 previously failed prompts.

### Conclusion
Through systematic data collection, user study, and automatic generation techniques, the paper comprehensively explored the threats of LLM jailbreak prompts and their countermeasures, providing an important reference for improving LLM security in the future.