Abstract:Vision transformers (ViTs) have witnessed significant progress in the past few years. Recently, the latest research revealed that ViTs are vulnerable to transfer-based attacks, in which attackers can use a local surrogate model to generate adversarial examples, then transfer these malicious examples to attack the target black-box ViT directly. Suffering from the threat of transfer-based attacks, it is challenging to deploy ViTs to security-critical tasks. Therefore, it becomes an exact need to explore the robustness of ViTs against transfer-based attacks. However, existing transfer-based attack methods do not fully consider the unique structure of ViT, and they indiscriminately attack the intermediate outputs token of ViTs, leading to the perturbations being focused on specific model information within the tokens, and further resulting in a limited transferability of the generated adversarial examples. To address the current limitations, we propose Token Importance Attack (TIA), a novel ViTs-oriented transfer-based attack method. Specifically, we introduce Randomly Shuffle Patches (RSP) strategy to expand the diversity of the input space. By applying RSP, we can generate multiple shuffled images from a single image, allowing us to obtain multiple token gradients. Then TIA ensembles these token gradients of shuffled images as a guide map to focus the perturbation on the model-independent information in the token rather than model-specific information. Benefiting from these two components, TIA can avoid overfitting to the surrogate model, thus enhancing the transferability of the crafted adversarial examples. Extensive experiments conducted on common datasets with different ViTs and CNNs have demonstrated the effectiveness of TIA.

Towards Transferable Adversarial Attacks on Image and Video Transformers

Protego: Detecting Adversarial Examples for Vision Transformers Via Intrinsic Capabilities

Towards transferable adversarial attacks on vision transformers for image classification

Generating Transferable Adversarial Examples Against Vision Transformers

On Improving Adversarial Transferability of Vision Transformers

Improving the Transferability of Adversarial Examples with Restructure Embedded Patches

Improving transferable adversarial attack for vision transformers via global attention and local drop

Transferable Adversarial Attack for Both Vision Transformers and Convolutional Networks Via Momentum Integrated Gradients

Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization

Adaptive Cross-Modal Transferable Adversarial Attacks From Images to Videos

Dual stage black-box adversarial attack against vision transformer

TIA: Token Importance Transferable Attack on Vision Transformers.

Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers

Query-Efficient Hard-Label Black-Box Attack against Vision Transformers

Enhancing Adversarial Transferability with Partial Blocks on Vision Transformer

Towards Efficient Adversarial Training on Vision Transformers

Transferable Adversarial Attacks for Image and Video Object Detection

Boosting the Transferability of Video Adversarial Examples Via Temporal Translation.

Channel-augmented Joint Transformation for Transferable Adversarial Attacks

A Target-oriented Approach to Enhancing the Transferability of Targeted Adversarial Attacks

Defending Backdoor Attacks on Vision Transformer via Patch Processing