Abstract:The emergence of vision transformers (ViTs) in image classification has shifted the methodologies for visual representation learning. In particular, ViTs learn visual representation at full receptive field per layer across all the image patches, in comparison to the increasing receptive fields of CNNs across layers and other alternatives (e.g., large kernels and atrous convolution). In this work, for the first time we explore the global context learning potentials of ViTs for dense visual prediction (e.g., semantic segmentation). Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information, critical for dense prediction tasks. We first demonstrate that encoding an image as a sequence of patches, a vanilla ViT without local convolution and resolution reduction can yield stronger visual representation for semantic segmentation. For example, our model, termed as SEgmentation TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the test leaderboard on the day of submission) and performs competitively on Cityscapes. However, the basic ViT architecture falls short in broader dense prediction applications, such as object detection and instance segmentation, due to its lack of a pyramidal structure, high computational demand, and insufficient local context. For tackling general dense visual prediction tasks in a cost-effective manner, we further formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture. Extensive experiments show that our methods achieve appealing performance on a variety of dense prediction tasks (e.g., object detection and instance segmentation and semantic segmentation) as well as image classification.

CViT: Continuous Vision Transformer for Operator Learning

SemiCVT: Semi-Supervised Convolutional Vision Transformer for Semantic Segmentation

ViTO: Vision Transformer-Operator

STC-ViT: Spatio Temporal Continuous Vision Transformer for Weather Forecasting

CRViT: Vision transformer advanced by causality and inductive bias for image recognition

Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

DctViT: Discrete Cosine Transform Meet Vision Transformers

Vision Transformer: Vit and its Derivatives

Continuous Spatiotemporal Transformers

CF-ViT: A General Coarse-to-Fine Method for Vision Transformer

MMViT: Multiscale Multiview Vision Transformers

Convolutional Embedding Makes Hierarchical Vision Transformer Stronger

CViT: A Convolution Vision Transformer for Video Abnormal Behavior Detection and Localization

MPViT: Multi-Path Vision Transformer for Dense Prediction

Multi-view Convolutional Vision Transformer for 3D Object Recognition.

Vision Transformers: From Semantic Segmentation to Dense Prediction

A Comprehensive Survey of Transformers for Computer Vision

LC2R-ViT: Long-Range Cross-Residual Vision Transformer for Medical Image Classification

A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

Supplementary Material: Continual Learning with Lifelong Vision Transformer

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias