Abstract:The use of transformers for vision tasks has challenged the traditional dominant role of convolutional neural networks (CNN) in computer vision (CV). For image classification tasks, Vision Transformer (ViT) effectively establishes spatial relationships between patches within images, directing attention to important areas for accurate predictions. However, similar to CNNs, ViTs are vulnerable to adversarial attacks, which mislead the image classifier into making incorrect decisions on images with carefully designed perturbations. Moreover, adversarial patch attacks, which introduce arbitrary perturbations within a small area, pose a more serious threat to ViTs. Even worse, traditional detection methods, originally designed for CNN models, are impractical or suffer significant performance degradation when applied to ViTs, and they generally overlook patch attacks. In this paper, we propose ViTGuard as a general detection method for defending ViT models against adversarial attacks, including typical attacks where perturbations spread over the entire input and patch attacks. ViTGuard uses a Masked Autoencoder (MAE) model to recover randomly masked patches from the unmasked regions, providing a flexible image reconstruction strategy. Then, threshold-based detectors leverage distinctive ViT features, including attention maps and classification (CLS) token representations, to distinguish between normal and adversarial samples. The MAE model does not involve any adversarial samples during training, ensuring the effectiveness of our detectors against unseen attacks. ViTGuard is compared with seven existing detection methods under nine attacks across three datasets. The evaluation results show the superiority of ViTGuard over existing detectors. Finally, considering the potential detection evasion, we further demonstrate ViTGuard's robustness against adaptive attacks for evasion.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the vulnerability of the Vision Transformer (ViT) model when facing adversarial sample attacks, especially against \(L_p\) -norm attacks and patch attacks. Specifically: 1. **Vulnerability of ViT to Adversarial Attacks**: Although the ViT model performs excellently in image classification tasks, like convolutional neural networks (CNN), they are vulnerable to adversarial attacks. Adversarial attacks mislead image classifiers to make wrong decisions by adding carefully - designed tiny perturbations to the input image. 2. **Deficiencies of Existing Detection Methods**: Most of the traditional adversarial sample detection methods are designed for CNNs and are ineffective or simply not feasible when applied to ViT. Moreover, these methods usually overlook the defense against patch attacks, which pose a greater threat to ViT. 3. **Lack of a Universal Detection Mechanism**: Currently, there is a lack of a universal mechanism that can effectively detect \(L_p\) -norm attacks and patch attacks without modifying or retraining the ViT model. To solve these problems, the paper proposes a new detection method named ViTGuard. ViTGuard utilizes the attention mechanism (attention maps) and classification (CLS) token representation unique to the ViT model to achieve the detection of adversarial samples through the following steps: - **Image Reconstruction**: Use the Masked Autoencoder (MAE) model to recover the original image from a partially occluded image. - **Feature Extraction**: Extract attention maps and CLS representations from the original image and the reconstructed image. - **Distance Calculation**: Calculate the \(L_2\) distance between the attention maps and CLS representations of the original image and the reconstructed image. If the distance exceeds a preset threshold, the input is considered an adversarial sample. The advantage of ViTGuard is that it does not require any adversarial samples for training and can flexibly respond to different types of adversarial attacks, including \(L_p\) -norm attacks and patch attacks. Experimental results show that ViTGuard outperforms existing detection methods on multiple datasets.

ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer

Protego: Detecting Adversarial Examples for Vision Transformers Via Intrinsic Capabilities

Enhancing the robustness of vision transformer defense against adversarial attacks based on squeeze-and-excitation module

Zero-Shot Certified Defense against Adversarial Patches with Vision Transformers

Defending Backdoor Attacks on Vision Transformer via Patch Processing

Are Vision Transformers Robust to Patch Perturbations?

Defending Against Universal Patch Attacks by Restricting Token Attention in Vision Transformers

Towards Efficient Adversarial Training on Vision Transformers

Generating Transferable Adversarial Examples Against Vision Transformers

Query-Efficient Hard-Label Black-Box Attack against Vision Transformers

Reveal of Vision Transformers Robustness against Adversarial Attacks

Improving Robustness for Vision Transformer with a Simple Dynamic Scanning Augmentation

On the Adversarial Robustness of Vision Transformers

Evaluating and enhancing the robustness of vision transformers against adversarial attacks in medical imaging

When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture

Dual stage black-box adversarial attack against vision transformer

Towards Practical Certifiable Patch Defense with Vision Transformer

Towards Transferable Adversarial Attacks on Image and Video Transformers

CSAdv: Class-Specific Adversarial Patches for DETR-Style Object Detection

Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers

Towards transferable adversarial attacks on vision transformers for image classification