Abstract:Vision transformer has achieved impressive performance for many vision tasks. However, it may suffer from high redundancy in capturing local features for shallow layers. Local self-attention or early-stage convolutions are thus utilized, which sacrifice the capacity to capture long-range dependency. A challenge then arises: can we access efficient and effective global context modeling at the early stages of a neural network? To address this issue, we draw inspiration from the design of superpixels, which reduces the number of image primitives in subsequent processing, and introduce super tokens into vision transformer. Super tokens attempt to provide a semantically meaningful tessellation of visual content, thus reducing the token number in self-attention as well as preserving global modeling. Specifically, we propose a simple yet strong super token attention (STA) mechanism with three steps: the first samples super tokens from visual tokens via sparse association learning, the second performs self-attention on super tokens, and the last maps them back to the original token space. STA decomposes vanilla global attention into multiplications of a sparse association map and a low-dimensional attention, leading to high efficiency in capturing global dependencies. Based on STA, we develop a hierarchical vision transformer. Extensive experiments demonstrate its strong performance on various vision tasks. In particular, without any extra training data or label, it achieves 86.4% top-1 accuracy on ImageNet-1K with less than 100M parameters. It also achieves 53.9 box AP and 46.8 mask AP on the COCO detection task, and 51.9 mIOU on the ADE20K semantic segmentation task. Code is released at <a class="link-external link-https" href="https://github.com/hhb072/STViT" rel="external noopener nofollow">this https URL</a>.

TTST: A Top-k Token Selective Transformer for Remote Sensing Image Super-Resolution

Cross-Spatial Pixel Integration and Cross-Stage Feature Fusion Based Transformer Network for Remote Sensing Image Super-Resolution

Activated Sparsely Sub-Pixel Transformer for Remote Sensing Image Super-Resolution

Enhanced Window-Based Self-Attention with Global and Multi-Scale Representations for Remote Sensing Image Super-Resolution

Vision Transformer with Super Token Sampling

An Efficient Hybrid CNN-Transformer Approach for Remote Sensing Super-Resolution

Remote Sensing Scene Classification via Second-Order Differentiable Token Transformer Network

Multi-Scale Super Token Transformer for Hyperspectral Image Classification

Hybrid-Scale Hierarchical Transformer for Remote Sensing Image Super-Resolution

A novel spatial and spectral transformer network for hyperspectral image super-resolution

DRCT: Saving Image Super-resolution away from Information Bottleneck

A Lightweight Distillation CNN-transformer Architecture for Remote Sensing Image Super-Resolution

A Spatial–Spectral Transformer for Hyperspectral Image Classification Based on Global Dependencies of Multi-Scale Features

Scale-Aware Backprojection Transformer for Single Remote Sensing Image Super-Resolution

Efficient Transformer for Single Image Super-Resolution.

DESAT: A Distance-Enhanced Strip Attention Transformer for Remote Sensing Image Super-Resolution

Spectral Superresolution Using Transformer with Convolutional Spectral Self-Attention

DsTer: A dense spectral transformer for remote sensing spectral super-resolution

Adaptive Spatial Tokenization Transformer for Salient Object Detection in Optical Remote Sensing Images.

Multi-granularity Backprojection Transformer for Remote Sensing Image Super-Resolution

DTCNet: Transformer-CNN Distillation for Super-Resolution of Remote Sensing Image