Abstract:Over the past few years, vision transformers (ViTs) have consistently demonstrated remarkable performance across various visual recognition tasks. However, attempts to enhance their robustness have yielded limited success, mainly focusing on different training strategies, input patch augmentation, or network structural enhancements. These approaches often involve extensive training and fine-tuning, which are time-consuming and resource-intensive. To tackle these obstacles, we introduce a novel approach named Spatial Autocorrelation Token Analysis (SATA). By harnessing spatial relationships between token features, SATA enhances both the representational capacity and robustness of ViT models. This is achieved through the analysis and grouping of tokens according to their spatial autocorrelation scores prior to their input into the Feed-Forward Network (FFN) block of the self-attention mechanism. Importantly, SATA seamlessly integrates into existing pre-trained ViT baselines without requiring retraining or additional fine-tuning, while concurrently improving efficiency by reducing the computational load of the FFN units. Experimental results show that the baseline ViTs enhanced with SATA not only achieve a new state-of-the-art top-1 accuracy on ImageNet-1K image classification (94.9%) but also establish new state-of-the-art performance across multiple robustness benchmarks, including ImageNet-A (top-1=63.6%), ImageNet-R (top-1=79.2%), and ImageNet-C (mCE=13.6%), all without requiring additional training or fine-tuning of baseline models.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to enhance the robustness and representational ability of Vision Transformers (ViTs) while improving their computational efficiency without additional training or fine - tuning. Specifically, most of the existing methods for improving ViT robustness rely on different training strategies, input patch augmentation, or network structure improvements. These methods usually require a large amount of training and fine - tuning, which are time - consuming and resource - intensive. To solve these problems, the author introduced a new method named **Spatial Autocorrelation Token Analysis (SATA)**. ### Main problems and solutions 1. **Limitations of existing methods**: - Existing methods such as data augmentation, contrastive learning strategies, and network structure adjustment have improved the performance and robustness of ViT to a certain extent, but they require a large amount of retraining or fine - tuning, especially on large - scale datasets (such as ImageNet - 1K, ImageNet - 21K). - These methods are not only time - consuming and resource - intensive, but may also sacrifice performance while improving robustness. 2. **Innovations of SATA**: - **Spatial autocorrelation analysis**: SATA processes tokens before inputting them into the feed - forward network (FFN) module of the self - attention mechanism by analyzing and grouping the spatial autocorrelation scores of tokens. - **Seamless integration**: SATA can be seamlessly integrated into existing pre - trained ViT baseline models without retraining or additional fine - tuning. - **Improvement in computational efficiency**: By reducing the computational load of FFN units, SATA not only enhances the robustness of ViT but also improves the inference efficiency. ### Experimental results The experimental results show that the enhanced ViT model performs well in multiple benchmark tests: - In the ImageNet - 1K image classification task, SATA - B* achieved a top - 1 accuracy of 94.9%. - In multiple robustness benchmark tests (such as ImageNet - A, ImageNet - R, ImageNet - C), SATA also achieved new state - of - the - art levels, for example, reaching 63.6% on ImageNet - A, 79.2% on ImageNet - R, and an mCE of 13.6% on ImageNet - C. ### Summary By introducing spatial autocorrelation analysis, SATA effectively solves the training and computational resource bottleneck problems in existing ViT robustness enhancement methods and significantly improves the robustness and computational efficiency of ViT.

SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers

Vision Transformer with Sparse Scan Prior

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

Making Vision Transformers Efficient from A Token Sparsification View

Robustifying Token Attention for Vision Transformers

Vision Transformer with Super Token Sampling

DAT++: Spatially Dynamic Vision Transformer with Deformable Attention

AdaViT: Adaptive Vision Transformers for Efficient Image Recognition

Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets

Improving Vision Transformers by Revisiting High-Frequency Components

Robustifying Vision Transformer without Retraining from Scratch by Test-Time Class-Conditional Feature Alignment

FasterViT: Fast Vision Transformers with Hierarchical Attention

Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations

Auto-scaling Vision Transformers without Training

CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications

Adaptive Token Sampling For Efficient Vision Transformers

DctViT: Discrete Cosine Transform Meet Vision Transformers

AT-SNN: Adaptive Tokens for Vision Transformer on Spiking Neural Network

ScalableViT: Rethinking the Context-Oriented Generalization of Vision Transformer.

SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet