Abstract:Vision Transformers (ViTs) have a radically different architecture with significantly less inductive bias than Convolutional Neural Networks. Along with the improvement in performance, security and robustness of ViTs are also of great importance to study. In contrast to many recent works that exploit the robustness of ViTs against adversarial examples, this paper investigates a representative causative attack, i.e., backdoor. We first examine the vulnerability of ViTs against various backdoor attacks and find that ViTs are also quite vulnerable to existing attacks. However, we observe that the clean-data accuracy and backdoor attack success rate of ViTs respond distinctively to patch transformations before the positional encoding. Then, based on this finding, we propose an effective method for ViTs to defend both patch-based and blending-based trigger backdoor attacks via patch processing. The performances are evaluated on several benchmark datasets, including CIFAR10, GTSRB, and TinyImageNet, which show the proposed novel defense is very successful in mitigating backdoor attacks for ViTs. To the best of our knowledge, this paper presents the first defensive strategy that utilizes a unique characteristic of ViTs against backdoor attacks. The paper will appear in the Proceedings of the AAAI'23 Conference. This work was initially submitted in November 2021 to CVPR'22, then it was re-submitted to ECCV'22. The paper was made public in June 2022. The authors sincerely thank all the referees from the Program Committees of CVPR'22, ECCV'22, and AAAI'23.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the defense ability of Vision Transformers (ViTs) against backdoor attacks. Specifically, the paper explores the vulnerability of ViTs when facing different types of backdoor attacks (such as patch - based trigger attacks and hybrid trigger attacks), and proposes a method to effectively defend against these attacks through image patch processing. The following are the main research objectives of the paper: 1. **Evaluate the vulnerability of ViTs to backdoor attacks**: - Researchers first evaluated the performance of ViTs when facing different types of backdoor attacks (such as patch - based trigger attacks and hybrid trigger attacks), and found that ViTs are also vulnerable to these attacks. 2. **Observe the response differences of ViTs before and after patch processing**: - The paper found that, before and after patch processing (such as randomly deleting patches and randomly shuffling the order of patches), ViTs showed significant differences in the accuracy of clean data and the success rate of backdoor attacks. This difference was not observed in Convolutional Neural Networks (CNNs). 3. **Propose a new defense method**: - Based on the above findings, researchers proposed a new defense strategy to detect and mitigate backdoor attacks through patch processing (PatchDrop and PatchShuffle). This method utilizes the frequency of prediction changes of ViTs when processing backdoor samples, compared with the frequency of changes when processing clean samples, thereby identifying malicious samples. ### Specific problem solutions **What problem does this paper attempt to solve?** The problem that this paper attempts to solve is the vulnerability of Vision Transformers (ViTs) when facing backdoor attacks, and proposes an effective defense method. Specifically, the paper aims to: - **Evaluate the vulnerability of ViTs to backdoor attacks**: Verify the performance of ViTs when facing different types of backdoor attacks through experiments. - **Discover the unique response characteristics of ViTs**: Observe that the responses of ViTs before and after patch processing are significantly different from those of CNNs, especially in terms of the accuracy of clean data and the success rate of backdoor attacks. - **Propose a new defense strategy**: Based on patch processing techniques (PatchDrop and PatchShuffle), propose a method that can effectively detect and mitigate backdoor attacks. Through these studies, the paper not only reveals the potential security problems of ViTs, but also provides a practical solution to enhance the security and robustness of ViTs in practical applications.

Defending Backdoor Attacks on Vision Transformer via Patch Processing

Protego: Detecting Adversarial Examples for Vision Transformers Via Intrinsic Capabilities

ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer

You Are Catching My Attention: Are Vision Transformers Bad Learners under Backdoor Attacks?

Zero-Shot Certified Defense against Adversarial Patches with Vision Transformers

Backdoor Attack Against Vision Transformers via Attention Gradient-Based Image Erosion

TrojViT: Trojan Insertion in Vision Transformers

Query-Efficient Hard-Label Black-Box Attack against Vision Transformers

Towards Transferable Adversarial Attacks on Image and Video Transformers

Towards Practical Certifiable Patch Defense with Vision Transformer

Defending Against Universal Patch Attacks by Restricting Token Attention in Vision Transformers

Generating Transferable Adversarial Examples Against Vision Transformers

Enhancing the robustness of vision transformer defense against adversarial attacks based on squeeze-and-excitation module

Are Vision Transformers Robust to Patch Perturbations?

Improving the Transferability of Adversarial Examples with Restructure Embedded Patches

Improving Robustness for Vision Transformer with a Simple Dynamic Scanning Augmentation

Towards transferable adversarial attacks on vision transformers for image classification

When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture

Dual stage black-box adversarial attack against vision transformer

Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers

Megatron: Evasive Clean-Label Backdoor Attacks against Vision Transformer