Abstract:The emergence of vision transformers (ViTs) in image classification has shifted the methodologies for visual representation learning. In particular, ViTs learn visual representation at full receptive field per layer across all the image patches, in comparison to the increasing receptive fields of CNNs across layers and other alternatives (e.g., large kernels and atrous convolution). In this work, for the first time we explore the global context learning potentials of ViTs for dense visual prediction (e.g., semantic segmentation). Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information, critical for dense prediction tasks. We first demonstrate that encoding an image as a sequence of patches, a vanilla ViT without local convolution and resolution reduction can yield stronger visual representation for semantic segmentation. For example, our model, termed as SEgmentation TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the test leaderboard on the day of submission) and performs competitively on Cityscapes. However, the basic ViT architecture falls short in broader dense prediction applications, such as object detection and instance segmentation, due to its lack of a pyramidal structure, high computational demand, and insufficient local context. For tackling general dense visual prediction tasks in a cost-effective manner, we further formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture. Extensive experiments show that our methods achieve appealing performance on a variety of dense prediction tasks (e.g., object detection and instance segmentation and semantic segmentation) as well as image classification.

Feature‐enhanced representation with transformers for multi‐view stereo

Transformer-guided Feature Pyramid Network for Multi-View Stereo

MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo

MTD-MVSNet: Multi-view Stereo Network with Multi-scale Transformer and Dual Attention

Modeling Long-Range Dependencies and Epipolar Geometry for Multi-View Stereo

MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth

MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth

CT-MVSNet: Curvature-guided multi-view stereo with transformers

MVSTER: Epipolar Transformer for Efficient Multi-View Stereo

MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer

A Global Depth-Range-Free Multi-View Stereo Transformer Network with Pose Embedding

Multi-View Stereo Representation Revist: Region-Aware MVSNet

When Epipolar Constraint Meets Non-local Operators in Multi-View Stereo

CostFormer:Cost Transformer for Cost Aggregation in Multi-view Stereo

Multi-view 3D Reconstruction with Transformer

MAFormer: A transformer network with multi-scale attention fusion for visual recognition

MMViT: Multiscale Multiview Vision Transformers

Multi-view 3D Reconstruction from Video with Transformer.

Vision Transformers: From Semantic Segmentation to Dense Prediction

Multiview Transformers for Video Recognition

Transformer-based stereo-aware 3D object detection from binocular images