Abstract:Vision transformer has achieved impressive performance for many vision tasks. However, it may suffer from high redundancy in capturing local features for shallow layers. Local self-attention or early-stage convolutions are thus utilized, which sacrifice the capacity to capture long-range dependency. A challenge then arises: can we access efficient and effective global context modeling at the early stages of a neural network? To address this issue, we draw inspiration from the design of superpixels, which reduces the number of image primitives in subsequent processing, and introduce super tokens into vision transformer. Super tokens attempt to provide a semantically meaningful tessellation of visual content, thus reducing the token number in self-attention as well as preserving global modeling. Specifically, we propose a simple yet strong super token attention (STA) mechanism with three steps: the first samples super tokens from visual tokens via sparse association learning, the second performs self-attention on super tokens, and the last maps them back to the original token space. STA decomposes vanilla global attention into multiplications of a sparse association map and a low-dimensional attention, leading to high efficiency in capturing global dependencies. Based on STA, we develop a hierarchical vision transformer. Extensive experiments demonstrate its strong performance on various vision tasks. In particular, without any extra training data or label, it achieves 86.4% top-1 accuracy on ImageNet-1K with less than 100M parameters. It also achieves 53.9 box AP and 46.8 mask AP on the COCO detection task, and 51.9 mIOU on the ADE20K semantic segmentation task. Code is released at <a class="link-external link-https" href="https://github.com/hhb072/STViT" rel="external noopener nofollow">this https URL</a>.

DenseDINO: Boosting Dense Self-Supervised Learning with Token-Based Point-Level Consistency.

SDPT: Semantic-Aware Dimension-Pooling Transformer for Image Segmentation

Vision Transformers for Dense Prediction

DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment

Upsampling DINOv2 features for unsupervised vision tasks and weakly supervised materials segmentation

Mask DINO: Towards a Unified Transformer-Based Framework for Object Detection and Segmentation

Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

DiT: Efficient Vision Transformers with Dynamic Token Routing

DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding

Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning

Vision Transformer with Super Token Sampling

UIFormer: A Unified Transformer-based Framework for Incremental Few-Shot Object Detection and Instance Segmentation

SDTP: Semantic-aware Decoupled Transformer Pyramid for Dense Image Prediction

$\texttt{DINO-Foresight}$: Looking into the Future with DINO

MST: Masked Self-Supervised Transformer for Visual Representation

Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation

DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video

Analyzing Local Representations of Self-supervised Vision Transformers

Change Dino: A Unified Transformer-Based Framework for Object-Level Change Detection and Segmentation in Remote Sensing Imagery

Dense Transformer Networks

A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence