Abstract:Contrastive vision-language representation learning has achieved state-of-the-art performance for zero-shot classification, by learning from millions of image-caption pairs crawled from the internet. However, the massive data that powers large multimodal models such as CLIP, makes them extremely vulnerable to various types of targeted data poisoning and backdoor attacks. Despite this vulnerability, robust contrastive vision-language pre-training against such attacks has remained unaddressed. In this work, we propose ROCLIP, the first effective method for robust pre-training multimodal vision-language models against targeted data poisoning and backdoor attacks. ROCLIP effectively breaks the association between poisoned image-caption pairs by considering a relatively large and varying pool of random captions, and matching every image with the text that is most similar to it in the pool instead of its own caption, every few <a class="link-external link-http" href="http://epochs.It" rel="external noopener nofollow">this http URL</a> also leverages image and text augmentations to further strengthen the defense and improve the performance of the model. Our extensive experiments show that ROCLIP renders state-of-the-art targeted data poisoning and backdoor attacks ineffective during pre-training CLIP models. In particular, ROCLIP decreases the success rate for targeted data poisoning attacks from 93.75% to 12.5% and that of backdoor attacks down to 0%, while improving the model's linear probe performance by 10% and maintains a similar zero shot performance compared to CLIP. By increasing the frequency of matching, ROCLIP is able to defend strong attacks, which add up to 1% poisoned examples to the data, and successfully maintain a low attack success rate of 12.5%, while trading off the performance on some tasks.

Adversarial Backdoor Defense in CLIP

BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning

CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning

Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threats

CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning

BDetCLIP: Multimodal Prompting Contrastive Test-Time Backdoor Detection

AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning

Model-Contrastive Learning for Backdoor Elimination

BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP

Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks

Backdoor Contrastive Learning via Bi-level Trigger Optimization

Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks

NCL: Textual Backdoor Defense Using Noise-augmented Contrastive Learning

Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning

Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP

Poisoning and Backdooring Contrastive Learning

Backdoor Attack on Unpaired Medical Image-Text Foundation Models: A Pilot Study on MedCLIP

On the Difficulty of Defending Contrastive Learning against Backdoor Attacks

MAE-MACD: the Masked Adversarial Contrastive Distillation Algorithm Grounded in Masked Autoencoders

BAGEL: Backdoor Attacks against Federated Contrastive Learning

CorruptEncoder: Data Poisoning based Backdoor Attacks to Contrastive Learning