Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training

Youliang Yuan,Wenxiang Jiao,Wenxuan Wang,Jen-tse Huang,Jiahao Xu,Tian Liang,Pinjia He,Zhaopeng Tu

2024-07-12

Abstract:This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) by identifying and tackling a refusal position bias within safety tuning data, which compromises the models' ability to appropriately refuse generating unsafe content. We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position, significantly enhancing their safety capabilities. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model families across six attack scenarios, demonstrates that our method not only improves model safety without compromising performance but also surpasses well-known models such as GPT-4 in defending against attacks. Importantly, our approach successfully defends recent advanced attack methods (e.g., CodeAttack) that have jailbroken GPT-4 and LLaMA3-70B-Instruct. Our code and data can be found at <a class="link-external link-https" href="https://github.com/RobustNLP/DeRTa" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

This paper focuses on the security issues of large language models (LLMs), particularly their ability to refuse unsafe content during generation. The research identifies a "refusal position bias" in the current security fine-tuning methods, where the model tends to refuse at the beginning of the response and overlooks the ability to identify and stop unsafe content during the generation process. To address this issue, the paper proposes a new method called "Decoupled Refusal Training" (DeRTa). DeRTa consists of two innovative components: 1. Maximum Likelihood Estimation (MLE) with toxic response prefixes: By appending a randomly sized toxic response at the beginning of a safe response, the model is trained to recognize and avoid unsafe content at any position. 2. Reinforced Transition Optimization (RTO): Enables the model to smoothly transition from potential harm to safe refusal at each position in the toxic response sequence. Through empirical evaluation, DeRTa improves model security without sacrificing performance and performs well in defending against attacks, including GPT-4. The paper emphasizes the effectiveness of this approach against advanced attack methods such as code attacks, which have successfully bypassed the protection of other models. In conclusion, the paper aims to enhance the capability of large language models to identify and refuse unsafe content during the generation process, thereby improving model security.

Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training

Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment

Robust LLM safeguarding via refusal feature adversarial training

Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models

Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing

Mitigating Unsafe Feedback with Learning Constraints

Don't Say No: Jailbreaking LLM by Suppressing Refusal

Refusing Safe Prompts for Multi-modal Large Language Models

ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming

Locking Down the Finetuned LLMs Safety

Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teaming

SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors

RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content

Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generation

Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level

DROJ: A Prompt-Driven Attack against Large Language Models

Refusal in Language Models Is Mediated by a Single Direction

Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinement

Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs

Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models

Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs