Abstract:With Vision-Language Pre-training (VLP) models demonstrating powerful multimodal interaction capabilities, the application scenarios of neural networks are no longer confined to unimodal domains but have expanded to more complex multimodal V+L downstream tasks. The security vulnerabilities of unimodal models have been extensively examined, whereas those of VLP models remain challenging. We note that in CV models, the understanding of images comes from annotated information, while VLP models are designed to learn image representations directly from raw text. Motivated by this discrepancy, we developed the Feature Guidance Attack (FGA), a novel method that uses text representations to direct the perturbation of clean images, resulting in the generation of adversarial images. FGA is orthogonal to many advanced attack strategies in the unimodal domain, facilitating the direct application of rich research findings from the unimodal to the multimodal scenario. By appropriately introducing text attack into FGA, we construct Feature Guidance with Text Attack (FGA-T). Through the interaction of attacking two modalities, FGA-T achieves superior attack effects against VLP models. Moreover, incorporating data augmentation and momentum mechanisms significantly improves the black-box transferability of FGA-T. Our method demonstrates stable and effective attack capabilities across various datasets, downstream tasks, and both black-box and white-box settings, offering a unified baseline for exploring the robustness of VLP models.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to explore the general architecture of adversarial attacks in Vision - Language Pretraining (VLP) models in order to understand the adversarial vulnerability of these models in multimodal tasks. Specifically, the researchers note that in computer vision (CV) models, the understanding of images mainly depends on annotation information; while in VLP models, the models directly learn image representations from the original text. Based on this difference, they developed a new method named Feature - Guided Attack (FGA), which uses text representations to guide perturbations of clean images, thereby generating adversarial images. In addition, by introducing text attacks, a method combining feature - guided and text attacks (FGA - T) was constructed, and by interacting with attacks on the two modalities, a better attack effect on VLP models was achieved. The study also explored how to directly apply the rich research results in the unimodal field to the multimodal scenario to bridge the gap between the two, and provided a unified benchmark for exploring the robustness of VLP models. The main contributions of the paper include: 1. Proposed FGA, using the original text as a supervision source, to perform adversarial attacks on VLP models, inducing the network to misinterpret adversarial images. 2. Introduced cross - modal interaction, forming a new multimodal adversarial attack through adversarial text, enhancing the white - box attack strength, and improving the black - box transferability through additional mechanisms. 3. This method is theoretically orthogonal to any unimodal attack enhancement mechanism. Experimental evidence based on multiple datasets and VLP models shows that this method is widely applicable to various V + L multimodal tasks, providing a unified baseline for exploring multimodal robustness.

A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models

Towards Adversarial Attack on Vision-Language Pre-training Models

Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach

One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models

Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective

VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models

Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Models

Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction

Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models

Universal Adversarial Perturbations for Vision-Language Pre-trained Models

Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Models

On Evaluating Adversarial Robustness of Large Vision-Language Models

Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks

Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning

Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Models

SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation

Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models

Highly Transferable Diffusion-based Unrestricted Adversarial Attack on Pre-trained Vision-Language Models

White-box Multimodal Jailbreaks Against Large Vision-Language Models

Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory

PG-Attack: A Precision-Guided Adversarial Attack Framework Against Vision Foundation Models for Autonomous Driving