Abstract:Fine-tuning large language models (LLMs) on additional datasets is often necessary to optimize them for specific downstream tasks. However, existing safety alignment measures, which restrict harmful behavior during inference, are insufficient to mitigate safety risks during fine-tuning. Alarmingly, fine-tuning with just 10 toxic sentences can make models comply with harmful instructions. We introduce SafetyLock, a novel alignment intervention method that maintains robust safety post-fine-tuning through efficient and transferable mechanisms. SafetyLock leverages our discovery that fine-tuned models retain similar safety-related activation representations to their base models. This insight enables us to extract what we term the Meta-SafetyLock, a set of safety bias directions representing key activation patterns associated with safe responses in the original model. We can then apply these directions universally to fine-tuned models to enhance their safety. By searching for activation directions across multiple token dimensions, SafetyLock achieves enhanced robustness and transferability. SafetyLock re-aligns fine-tuned models in under 0.01 seconds without additional computational cost. Our experiments demonstrate that SafetyLock can reduce the harmful instruction response rate from 60% to below 1% in toxic fine-tuned models. It surpasses traditional methods in both performance and efficiency, offering a scalable, non-invasive solution for ensuring the safety of customized LLMs. Our analysis across various fine-tuning scenarios confirms SafetyLock's robustness, advocating its integration into safety protocols for aligned LLMs. The code is released at <a class="link-external link-https" href="https://github.com/zhu-minjun/SafetyLock" rel="external noopener nofollow">this https URL</a>.

Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching

Superficial Safety Alignment Hypothesis

Safety Alignment Should Be Made More Than Just a Few Tokens Deep

A safety realignment framework via subspace-oriented model fusion for large language models

Finding Safety Neurons in Large Language Models

Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models

SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Tasks

Locking Down the Finetuned LLMs Safety

Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements

Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging

Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations

Safety Alignment for Vision Language Models

SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance

ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectors

Robustifying Safety-Aligned Large Language Models through Clean Data Curation

Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack

Safety Layers in Aligned Large Language Models: The Key to LLM Security

ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming

Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications