Abstract:The use of transformers for vision tasks has challenged the traditional dominant role of convolutional neural networks (CNN) in computer vision (CV). For image classification tasks, Vision Transformer (ViT) effectively establishes spatial relationships between patches within images, directing attention to important areas for accurate predictions. However, similar to CNNs, ViTs are vulnerable to adversarial attacks, which mislead the image classifier into making incorrect decisions on images with carefully designed perturbations. Moreover, adversarial patch attacks, which introduce arbitrary perturbations within a small area, pose a more serious threat to ViTs. Even worse, traditional detection methods, originally designed for CNN models, are impractical or suffer significant performance degradation when applied to ViTs, and they generally overlook patch attacks. In this paper, we propose ViTGuard as a general detection method for defending ViT models against adversarial attacks, including typical attacks where perturbations spread over the entire input and patch attacks. ViTGuard uses a Masked Autoencoder (MAE) model to recover randomly masked patches from the unmasked regions, providing a flexible image reconstruction strategy. Then, threshold-based detectors leverage distinctive ViT features, including attention maps and classification (CLS) token representations, to distinguish between normal and adversarial samples. The MAE model does not involve any adversarial samples during training, ensuring the effectiveness of our detectors against unseen attacks. ViTGuard is compared with seven existing detection methods under nine attacks across three datasets. The evaluation results show the superiority of ViTGuard over existing detectors. Finally, considering the potential detection evasion, we further demonstrate ViTGuard's robustness against adaptive attacks for evasion.

Robustness analysis of Visual Transformer based on adversarial attacks

Protego: Detecting Adversarial Examples for Vision Transformers Via Intrinsic Capabilities

On the Adversarial Robustness of Vision Transformers

Enhancing the robustness of vision transformer defense against adversarial attacks based on squeeze-and-excitation module

Reveal of Vision Transformers Robustness against Adversarial Attacks

When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture

Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem

A Light Recipe to Train Robust Vision Transformers

Attacking Transformers with Feature Diversity Adversarial Perturbation

Dual stage black-box adversarial attack against vision transformer

Are Vision Transformers Robust to Patch Perturbations?

Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models

ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer

Evaluating and enhancing the robustness of vision transformers against adversarial attacks in medical imaging

Deeper Insights into the Robustness of ViTs towards Common Corruptions

On Improving Adversarial Transferability of Vision Transformers

Query-Efficient Hard-Label Black-Box Attack against Vision Transformers

On the unreasonable vulnerability of transformers for image restoration -- and an easy fix

Generating Transferable Adversarial Examples Against Vision Transformers

Decision-based Black-box Attack Against Vision Transformers Via Patch-wise Adversarial Removal

VTFR-AT: Adversarial Training with Visual Transformation and Feature Robustness