Abstract:Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to automatically discover prompts that can trigger generative models (such as language models) to produce unsafe responses. Specifically, the paper proposes a method named Gradient - Based Red Teaming (GBRT), aiming to automatically generate diverse prompts that are likely to cause language models to generate unsafe responses. This method is crucial for model alignment and evaluation, but traditionally, red - teaming tests conducted by humans are labor - intensive and difficult to scale. By automating this process, the GBRT method aims to improve efficiency and effectiveness, especially when the language model is fine - tuned to reduce unsafe outputs. The core contributions of the paper are as follows: 1. **Automatic Prompt Generation**: GBRT learns to generate prompts that can make the language model generate unsafe responses. This is achieved by taking the probability distribution of prompts as input and using a safety classifier to evaluate the safety of responses. 2. **Gradient Back - Propagation**: To update the prompts, GBRT utilizes the gradients of the safety classifier and the language model, and updates the probability distribution of the prompts through back - propagation to minimize the safety score. 3. **Enhanced Realism**: To generate more realistic prompts, the paper proposes two variants, one is to add a realism loss, and the other is to fine - tune a pre - trained model to generate prompts instead of directly learning the prompts. 4. **Experimental Verification**: The experimental results show that GBRT is more effective in discovering prompts that can trigger language models to generate unsafe responses than reinforcement - learning - based red - teaming test methods, even when the language model is fine - tuned to reduce unsafe outputs. Overall, this paper aims to improve the safety and reliability of generative models through automating the red - teaming test process, especially in high - risk fields such as medicine and education.

Gradient-Based Language Model Red Teaming

Learning diverse attacks on large language models for robust red-teaming and safety tuning

ASTPrompter: Weakly Supervised Automated Language Model Red-Teaming to Identify Likely Toxic Prompts

Curiosity-driven Red-teaming for Large Language Models

Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming

Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games

Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases

Automated Progressive Red Teaming

FLRT: Fluent Student-Teacher Redteaming

Explore, Establish, Exploit: Red Teaming Language Models from Scratch

Automated Red Teaming with GOAT: the Generative Offensive Agent Tester

Attack Prompt Generation for Red Teaming and Defending Large Language Models

Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?

RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent

Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts

Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?

AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

FLIRT: Feedback Loop In-context Red Teaming

ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models