Abstract:The emergence of vision transformers (ViTs) in image classification has shifted the methodologies for visual representation learning. In particular, ViTs learn visual representation at full receptive field per layer across all the image patches, in comparison to the increasing receptive fields of CNNs across layers and other alternatives (e.g., large kernels and atrous convolution). In this work, for the first time we explore the global context learning potentials of ViTs for dense visual prediction (e.g., semantic segmentation). Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information, critical for dense prediction tasks. We first demonstrate that encoding an image as a sequence of patches, a vanilla ViT without local convolution and resolution reduction can yield stronger visual representation for semantic segmentation. For example, our model, termed as SEgmentation TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the test leaderboard on the day of submission) and performs competitively on Cityscapes. However, the basic ViT architecture falls short in broader dense prediction applications, such as object detection and instance segmentation, due to its lack of a pyramidal structure, high computational demand, and insufficient local context. For tackling general dense visual prediction tasks in a cost-effective manner, we further formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture. Extensive experiments show that our methods achieve appealing performance on a variety of dense prediction tasks (e.g., object detection and instance segmentation and semantic segmentation) as well as image classification.

Transfer Vision Patterns for Multi-Task Pixel Learning.

Tube-Embedded Transformer for Pixel Prediction

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

AutoTaskFormer: Searching Vision Transformers for Multi-task Learning

InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understanding

Vision Pair Learning: An Efficient Training Framework for Image Classification

Vision Transformer Adapters for Generalizable Multitask Learning

P2T: Pyramid Pooling Transformer for Scene Understanding

Multimodal Token Fusion for Vision Transformers

A Bio-Inspired Visual Perception Transformer for Cross-Domain Semantic Segmentation of High-Resolution Remote Sensing Images

Efficient Computation Sharing for Multi-Task Visual Scene Understanding

GPPF: A General Perception Pre-training Framework via Sparsely Activated Multi-Task Learning

Aggregated Pyramid Vision Transformer: Split-transform-merge Strategy for Image Recognition without Convolutions

Brain-Inspired Stepwise Patch Merging for Vision Transformers

Correlated Attribute Transfer with Multi-Task Graph-Guided Fusion.

PointMTL: Multi-Transform Learning for Effective 3D Point Cloud Representations

Vision Transformers: From Semantic Segmentation to Dense Prediction

Visual Interest Prediction with Attentive Multi-Task Transfer Learning

Vision Transformers with Patch Diversification

MPViT: Multi-Path Vision Transformer for Dense Prediction

PTET: A progressive token exchanging transformer for infrared and visible image fusion