Abstract:As a de facto solution, the vanilla Vision Transformers (ViTs) are encouraged to model long-range dependencies between arbitrary image patches while the global attended receptive field leads to quadratic computational cost. Another branch of Vision Transformers exploits local attention inspired by CNNs, which only models the interactions between patches in small neighborhoods. Although such a solution reduces the computational cost, it naturally suffers from small attended receptive fields, which may limit the performance. In this work, we explore effective Vision Transformers to pursue a preferable trade-off between the computational complexity and size of the attended receptive field. By analyzing the patch interaction of global attention in ViTs, we observe two key properties in the shallow layers, namely locality and sparsity, indicating the redundancy of global dependency modeling in shallow layers of ViTs. Accordingly, we propose Multi-Scale Dilated Attention (MSDA) to model local and sparse patch interaction within the sliding window. With a pyramid architecture, we construct a Multi-Scale Dilated Transformer (DilateFormer) by stacking MSDA blocks at low-level stages and global multi-head self-attention blocks at high-level stages. Our experiment results show that our DilateFormer achieves state-of-the-art performance on various vision tasks. On ImageNet-1K classification task, DilateFormer achieves comparable performance with 70% fewer FLOPs compared with existing state-of-the-art models. Our DilateFormer-Base achieves 85.6% top-1 accuracy on ImageNet-1K classification task, 53.5% box mAP/46.1% mask mAP on COCO object detection/instance segmentation task and 51.1% MS mIoU on ADE20K semantic segmentation task.

Retina Vision Transformer (RetinaViT): Introducing Scaled Patches into Vision Transformers

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Scaling Vision Transformers

CF-ViT: A General Coarse-to-Fine Method for Vision Transformer

Improving Vision Transformers by Revisiting High-Frequency Components

Super Vision Transformer

A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation

ScopeViT: Scale-aware Vision Transformer

Scalable Vision Transformers with Hierarchical Pooling.

DctViT: Discrete Cosine Transform Meet Vision Transformers

FDViT: Improve the Hierarchical Architecture of Vision Transformer.

Vision Transformer: Vit and its Derivatives

ViTAR: Vision Transformer with Any Resolution

DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition

Vision Transformer with Progressive Sampling

ReViT: Enhancing Vision Transformers Feature Diversity with Attention Residual Connections

Slicing Vision Transformer for Flexible Inference

DeepViT: Towards Deeper Vision Transformer

Improving Vision Transformers for Incremental Learning

HSViT: Horizontally Scalable Vision Transformer

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers