Abstract:The emergence of vision transformers (ViTs) in image classification has shifted the methodologies for visual representation learning. In particular, ViTs learn visual representation at full receptive field per layer across all the image patches, in comparison to the increasing receptive fields of CNNs across layers and other alternatives (e.g., large kernels and atrous convolution). In this work, for the first time we explore the global context learning potentials of ViTs for dense visual prediction (e.g., semantic segmentation). Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information, critical for dense prediction tasks. We first demonstrate that encoding an image as a sequence of patches, a vanilla ViT without local convolution and resolution reduction can yield stronger visual representation for semantic segmentation. For example, our model, termed as SEgmentation TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the test leaderboard on the day of submission) and performs competitively on Cityscapes. However, the basic ViT architecture falls short in broader dense prediction applications, such as object detection and instance segmentation, due to its lack of a pyramidal structure, high computational demand, and insufficient local context. For tackling general dense visual prediction tasks in a cost-effective manner, we further formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture. Extensive experiments show that our methods achieve appealing performance on a variety of dense prediction tasks (e.g., object detection and instance segmentation and semantic segmentation) as well as image classification.

Local-to-Global Self-Attention in Vision Transformers

PLG-ViT: Vision Transformer with Parallel Local and Global Self-Attention

Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

Hierarchical local global transformer for point clouds analysis

Vision Transformers with Hierarchical Attention

Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight

Lite Vision Transformer with Enhanced Self-Attention

BOAT: Bilateral Local Attention Vision Transformer

On the Connection between Local Attention and Dynamic Depth-wise Convolution

HA-Transformer: Harmonious aggregation from local to global for object detection

Semantic-Aware Local-Global Vision Transformer

Unifying Global-Local Representations in Salient Object Detection with Transformer

Vision Transformers: From Semantic Segmentation to Dense Prediction

RegionViT: Regional-to-Local Attention for Vision Transformers

Transformers Meet Visual Learning Understanding: A Comprehensive Review

Glance-and-Gaze Vision Transformer

Conformer: Local Features Coupling Global Representations for Visual Recognition

Global–local transformer for single-image rain removal

ViT-LSLA: Vision Transformer with Light Self-Limited-Attention

Improving Vision Transformers by Overlapping Heads in Multi-Head Self-Attention

Light-Weight Vision Transformer with Parallel Local and Global Self-Attention