Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

Mikayel Samvelyan,Sharath Chandra Raparthy,Andrei Lupu,Eric Hambro,Aram H. Markosyan,Manish Bhatt,Yuning Mao,Minqi Jiang,Jack Parker-Holder,Jakob Foerster,Tim Rocktäschel,Roberta Raileanu

2024-07-23

Abstract:As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem, and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that fine-tuning models with synthetic data generated by the Rainbow Teaming method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

This paper aims to address the robustness issue of large language models (LLMs) when faced with adversarial prompts. Specifically, existing methods have the following limitations in identifying adversarial prompts: domain-specific focus, lack of diversity, or requiring extensive manual annotation. To address these issues, the researchers propose Rainbow Teaming, a novel black-box approach capable of generating diverse adversarial prompts. Rainbow Teaming views adversarial prompt generation as a Quality-Diversity (QD) problem and utilizes open-ended search to generate prompts that are both effective and diverse. The researchers conducted experiments using Rainbow Teaming on various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. The results show that Rainbow Teaming can reveal hundreds of effective adversarial prompts, with an attack success rate of over 90%. Moreover, fine-tuning the models with synthetic data generated by Rainbow Teaming can significantly enhance model security without sacrificing their general performance or helpfulness. Beyond the security domain, the researchers also demonstrate the potential applications of Rainbow Teaming in other areas such as question answering and cybersecurity, proving its broad applicability as a comprehensive tool for diagnosing and enhancing the robustness and reliability of LLMs.

Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique

Gradient-Based Language Model Red Teaming

Automated Progressive Red Teaming

AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

ASTPrompter: Weakly Supervised Automated Language Model Red-Teaming to Identify Likely Toxic Prompts

Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming

Learning diverse attacks on large language models for robust red-teaming and safety tuning

Curiosity-driven Red-teaming for Large Language Models

Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

Attack Prompt Generation for Red Teaming and Defending Large Language Models

Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts

PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

Robustness-aware Automatic Prompt Optimization

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?

DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints

FLRT: Fluent Student-Teacher Redteaming

Improved Generation of Adversarial Examples Against Safety-aligned LLMs

Universal and Transferable Adversarial Attacks on Aligned Language Models

Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation