Abstract:As the integration of the Large Language Models (LLMs) into various applications increases, so does their susceptibility to misuse, raising significant security concerns. Numerous jailbreak attacks have been proposed to assess the security defense of LLMs. Current jailbreak attacks mainly rely on scenario camouflage, prompt obfuscation, prompt optimization, and prompt iterative optimization to conceal malicious prompts. In particular, sequential prompt chains in a single query can lead LLMs to focus on certain prompts while ignoring others, facilitating context manipulation. This paper introduces SequentialBreak, a novel jailbreak attack that exploits this vulnerability. We discuss several scenarios, not limited to examples like Question Bank, Dialog Completion, and Game Environment, where the harmful prompt is embedded within benign ones that can fool LLMs into generating harmful responses. The distinct narrative structures of these scenarios show that SequentialBreak is flexible enough to adapt to various prompt formats beyond those discussed. Extensive experiments demonstrate that SequentialBreak uses only a single query to achieve a substantial gain of attack success rate over existing baselines against both open-source and closed-source models. Through our research, we highlight the urgent need for more robust and resilient safeguards to enhance LLM security and prevent potential misuse. All the result files and website associated with this research are available in this GitHub repository: <a class="link-external link-https" href="https://anonymous.4open.science/r/JailBreakAttack-4F3B/" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### The problems the paper attempts to solve This paper aims to explore how to bypass the security protection mechanisms of large language models (LLMs) by embedding malicious prompts into sequential prompt chains. Specifically, the paper introduces a new attack method named SequentialBreak, which exploits the vulnerability of uneven attention distribution that may exist in LLMs when processing multiple consecutive prompts. In this way, malicious prompts can be hidden among a series of seemingly harmless prompts, thus inducing LLMs to generate harmful responses. ### Main problem background With the wide use of large language models in various application scenarios, the potential risks of abuse are also increasing day by day. In order to deal with these risks, much research has focused on designing and evaluating new "jailbreak" attack methods, which attempt to bypass the security protection measures of LLMs to generate content that violates ethical norms. Existing jailbreak attack methods mainly rely on techniques such as scenario camouflage, prompt confusion, prompt optimization, and iterative optimization, but these methods usually require multiple queries and are easily detected. ### Main features of SequentialBreak 1. **Single - query**: SequentialBreak can achieve a high - success - rate attack with only one query. 2. **Black - box access**: This attack method only requires black - box access rights and does not need to understand the internal structure of the model. 3. **High adaptability**: This method can adapt to different prompt narrative structures and is suitable for multiple scenarios. 4. **Resource - efficient**: As a one - time attack, SequentialBreak is more efficient than existing methods and supports transfer learning. ### Attack strategy 1. **Template generation**: The attacker first generates a template, which is used to embed malicious prompts into a seemingly harmless context. 2. **Template selection**: Select a template suitable for the current attack scenario from the predefined templates. 3. **Select malicious target prompt**: Create or select a malicious target prompt, which aims to circumvent the LLMs' defense against harmful content. 4. **Template - specific formatting**: Format the malicious prompt so that it can be seamlessly integrated into the selected template. 5. **User prompt generation**: Embed the formatted malicious prompt into the template to form the final user prompt. 6. **Submit to the target LLM and analyze the response**: Submit the reconstructed prompt to the target LLM and analyze the response it generates to determine whether harmful content has been successfully generated. ### Experimental results The paper experimentally evaluated the attack effect of SequentialBreak in different scenarios, including question banks, dialogue completion, and game environments. The experimental results show that SequentialBreak exhibits a relatively high attack success rate on a variety of open - source and closed - source LLMs. In particular, in the dialogue completion scenario, some templates perform particularly well. ### Conclusion By introducing SequentialBreak, the paper reveals the potential security vulnerabilities of LLMs when processing sequential prompts and emphasizes the urgency of strengthening the security protection mechanisms of LLMs. This not only helps to improve the security of LLMs but also can prevent potential abuse behaviors.

SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains

"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

Multi-round jailbreak attack on large language models

Comprehensive Assessment of Jailbreak Attacks Against LLMs

Effective and Evasive Fuzz Testing-Driven Jailbreaking Attacks against LLMs

Jailbreak Attacks and Defenses Against Large Language Models: A Survey

JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models

ObscurePrompt: Jailbreaking Large Language Models via Obscure Input

Jailbreaking Black Box Large Language Models in Twenty Queries

What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks

JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks

DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers

Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring

Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks

SoK: Prompt Hacking of Large Language Models

Jailbreaking Proprietary Large Language Models using Word Substitution Cipher

SQL Injection Jailbreak: a structural disaster of large language models

DROJ: A Prompt-Driven Attack against Large Language Models

Distract Large Language Models for Automatic Jailbreak Attack

Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks