Abstract:Although fully convolution networks (FCNs) have dominated dense prediction tasks (e.g., semantic seg-mentation, depth estimation and object detection) for decades, they are inherently limited in captur-ing long-range structured relationship with the layers of local kernels. While recent Transformer-based models have proven extremely successful in computer vision tasks by capturing global representation, they would deteriorate dense prediction results by over-smoothing the regions containing fine details (e.g., boundaries and small objects). To this end, we aim to provide an alternative perspective by re-thinking local and global feature representation for the dense prediction task. Specifically, we deploy a Dual-Stream Convolution-Transformer architecture, called DSCT, by taking advantage of both the convolu-tion and Transformer to learn a rich feature representation, combining with a task decoder to provide a powerful dense prediction model. DSCT extracts high resolution local feature representation from convo-lution layers and global feature representation from Transformer layers. With the local and global context modeled explicitly in every layer, the two streams can be combined with a decoder to perform task of se-mantic segmentation, monocular depth estimation or object detection. Extensive experiments show that DSCT can achieve superior performance on the three tasks above. For semantic segmentation, DSCT builds a new state of the art on Cityscapes validation set (83.31% mIoU) with only 80,0 0 0 training iterations and appealing performance (49.27% mIoU) on ADE20K validation set, outperforming most of the alternatives. For monocular depth estimation, our model achieves 2.423 RMSE on KITTI Eigen split, superior to most of the convolution or Transformer counterparts. For object detection, without using FPN, we can achieve 44.5% APb on COCO dataset when using Faster R-CNN, which is higher than Conformer.(c) 2022 Elsevier Ltd. All rights reserved.

Conformer: Local Features Coupling Global Representations for Recognition and Detection

Conformer: Local Features Coupling Global Representations for Visual Recognition

LGFCTR: Local and Global Feature Convolutional Transformer for Image Matching

Combining transformer global and local feature extraction for object detection

ConvFormer: Plug-and-Play CNN-Style Transformers for Improving Medical Image Segmentation

D-CONFORMER: Deformable Sparse Transformer Augmented Convolution for Voxel-Based 3D Object Detection

Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition

Unifying Global-Local Representations in Salient Object Detection with Transformers

Global-Local Representation Coupling Network for Remote Sensing Image Change Detection

Unifying Global-Local Representations in Salient Object Detection with Transformer

Mixed-former: multi-fusion remote sensing change detection

CTIF-Net: A CNN-Transformer Iterative Fusion Network for Salient Object Detection

ConTNet: Why not use convolution and transformer at the same time?

ConvTransNet: A CNN–Transformer Network for Change Detection With Multiscale Global–Local Representations

Rethinking Local and Global Feature Representation for Dense Prediction

PCformer: A Parallel Convolutional Transformer Network for 360° Depth Estimation

Hybrid Token Transformer for Deep Face Recognition

Fusion of Dilated Convolution in CNN and Transformer Networks for Hyperspectral Image Classification

Dynamic multi-headed self-attention and multiscale enhancement vision transformer for object detection

Combining transformers with CNN for multi-focus image fusion

CONMW TRANSFORMER: A GENERAL VISION TRANSFORMER BACKBONE WITH MERGED-WINDOW ATTENTION