Abstract:The use of transformers for vision tasks has challenged the traditional dominant role of convolutional neural networks (CNN) in computer vision (CV). For image classification tasks, Vision Transformer (ViT) effectively establishes spatial relationships between patches within images, directing attention to important areas for accurate predictions. However, similar to CNNs, ViTs are vulnerable to adversarial attacks, which mislead the image classifier into making incorrect decisions on images with carefully designed perturbations. Moreover, adversarial patch attacks, which introduce arbitrary perturbations within a small area, pose a more serious threat to ViTs. Even worse, traditional detection methods, originally designed for CNN models, are impractical or suffer significant performance degradation when applied to ViTs, and they generally overlook patch attacks. In this paper, we propose ViTGuard as a general detection method for defending ViT models against adversarial attacks, including typical attacks where perturbations spread over the entire input and patch attacks. ViTGuard uses a Masked Autoencoder (MAE) model to recover randomly masked patches from the unmasked regions, providing a flexible image reconstruction strategy. Then, threshold-based detectors leverage distinctive ViT features, including attention maps and classification (CLS) token representations, to distinguish between normal and adversarial samples. The MAE model does not involve any adversarial samples during training, ensuring the effectiveness of our detectors against unseen attacks. ViTGuard is compared with seven existing detection methods under nine attacks across three datasets. The evaluation results show the superiority of ViTGuard over existing detectors. Finally, considering the potential detection evasion, we further demonstrate ViTGuard's robustness against adaptive attacks for evasion.

Privacy-Preserving Image Classification Using Vision Transformer

Privacy-Preserving Vision Transformer Using Images Encrypted with Restricted Random Permutation Matrices

Protego: Detecting Adversarial Examples for Vision Transformers Via Intrinsic Capabilities

Privacy-Preserving Vision Transformer on Permutation-Encrypted Images

Privacy-Preserving Image Classification Using Isotropic Network

Efficient Fine-Tuning with Domain Adaptation for Privacy-Preserving Vision Transformer

Disposable-key-based image encryption for collaborative learning of Vision Transformer

Effects of JPEG Compression on Vision Transformer Image Classification for Encryption-then-Compression Images

Vision Transformers for Remote Sensing Image Classification

ViT Enhanced Privacy-Preserving Secure Medical Data Sharing and Classification

ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer

Privacy-Preserving Image Retrieval Based on Disordered Local Histograms and Vision Transformer in Cloud Computing

Vision Transformers with Mixed-Resolution Tokenization

Enhanced Security with Encrypted Vision Transformer in Federated Learning

Implementing vision transformer for classifying 2D biomedical images

Human-imperceptible, Machine-recognizable Images

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition

Towards End-to-End Image Compression and Analysis with Transformers

Vision Transformer with Progressive Sampling