Abstract:Vision transformer has achieved impressive performance for many vision tasks. However, it may suffer from high redundancy in capturing local features for shallow layers. Local self-attention or early-stage convolutions are thus utilized, which sacrifice the capacity to capture long-range dependency. A challenge then arises: can we access efficient and effective global context modeling at the early stages of a neural network? To address this issue, we draw inspiration from the design of superpixels, which reduces the number of image primitives in subsequent processing, and introduce super tokens into vision transformer. Super tokens attempt to provide a semantically meaningful tessellation of visual content, thus reducing the token number in self-attention as well as preserving global modeling. Specifically, we propose a simple yet strong super token attention (STA) mechanism with three steps: the first samples super tokens from visual tokens via sparse association learning, the second performs self-attention on super tokens, and the last maps them back to the original token space. STA decomposes vanilla global attention into multiplications of a sparse association map and a low-dimensional attention, leading to high efficiency in capturing global dependencies. Based on STA, we develop a hierarchical vision transformer. Extensive experiments demonstrate its strong performance on various vision tasks. In particular, without any extra training data or label, it achieves 86.4% top-1 accuracy on ImageNet-1K with less than 100M parameters. It also achieves 53.9 box AP and 46.8 mask AP on the COCO detection task, and 51.9 mIOU on the ADE20K semantic segmentation task. Code is released at <a class="link-external link-https" href="https://github.com/hhb072/STViT" rel="external noopener nofollow">this https URL</a>.

A Stable Vision Transformer for Out-of-Distribution Generalization.

Delving Deep into the Generalization of Vision Transformers under Distribution Shifts

PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection

Super Vision Transformer

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

Vision Transformer with Progressive Sampling

CF-ViT: A General Coarse-to-Fine Method for Vision Transformer

Self-Distilled Vision Transformer for Domain Generalization

Denoising Vision Transformers

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Making Vision Transformers Efficient from A Token Sparsification View

Vision Transformer with Super Token Sampling

A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation

Effective Vision Transformer Training: A Data-Centric Perspective

Boosting Vanilla Lightweight Vision Transformers Via Re-parameterization

Vision Transformers with Patch Diversification

TPC-ViT: Token Propagation Controller for Efficient Vision Transformer

ViR:the Vision Reservoir

Improving Vision Transformers by Revisiting High-Frequency Components

ScalableViT: Rethinking the Context-Oriented Generalization of Vision Transformer.

Vision Transformer with Sparse Scan Prior