Abstract:Large Vision-Language Models (LVLMs), trained on multimodal big datasets, have significantly advanced AI by excelling in vision-language tasks. However, these models remain vulnerable to adversarial attacks, particularly jailbreak attacks, which bypass safety protocols and cause the model to generate misleading or harmful responses. This vulnerability stems from both the inherent susceptibilities of LLMs and the expanded attack surface introduced by the visual modality. We propose Sim-CLIP+, a novel defense mechanism that adversarially fine-tunes the CLIP vision encoder by leveraging a Siamese architecture. This approach maximizes cosine similarity between perturbed and clean samples, facilitating resilience against adversarial manipulations. Sim-CLIP+ offers a plug-and-play solution, allowing seamless integration into existing LVLM architectures as a robust vision encoder. Unlike previous defenses, our method requires no structural modifications to the LVLM and incurs minimal computational overhead. Sim-CLIP+ demonstrates effectiveness against both gradient-based adversarial attacks and various jailbreak techniques. We evaluate Sim-CLIP+ against three distinct jailbreak attack strategies and perform clean evaluations using standard downstream datasets, including COCO for image captioning and OKVQA for visual question answering. Extensive experiments demonstrate that Sim-CLIP+ maintains high clean accuracy while substantially improving robustness against both gradient-based adversarial attacks and jailbreak techniques. Our code and robust vision encoders are available at <a class="link-external link-https" href="https://github.com/speedlab-git/Robust-Encoder-against-Jailbreak-attack.git" rel="external noopener nofollow">this https URL</a>.

Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models

White-box Multimodal Jailbreaks Against Large Vision-Language Models

Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models

Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt

Jailbreak Large Vision-Language Models Through Multi-Modal Linkage

Visual Adversarial Examples Jailbreak Aligned Large Language Models

Efficient LLM-Jailbreaking by Introducing Visual Modality

JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks

Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models

IDEATOR: Jailbreaking Large Vision-Language Models Using Themselves

IDEATOR: Jailbreaking VLMs Using VLMs

Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks

Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models

Jailbreaking Attack against Multimodal Large Language Model

Query-Relevant Images Jailbreak Large Multi-Modal Models

Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles

ImgTrojan: Jailbreaking Vision-Language Models with ONE Image

Weak-to-Strong Jailbreaking on Large Language Models

Emerging Vulnerabilities in Frontier Models: Multi-Turn Jailbreak Attacks

Towards Adversarial Attack on Vision-Language Pre-training Models