Abstract:Recently, Vision Transformer and its variants have demonstrated remarkable performance on various computer vision tasks, thanks to its competence in capturing global visual dependencies through self-attention. However, global self-attention suffers from high computational cost due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection and semantic segmentation). Many recent works have attempted to reduce the cost by applying fine-grained local attention, but these approaches cripple the long-range modeling power of the original self-attention mechanism. Furthermore, these approaches usually have similar receptive fields within each layer, thus limiting the ability of each self-attention layer to capture multi-scale features, resulting in performance degradation when handling images with objects of different scales. To address these issues, we develop the Diagonal-shaped Window (DiagSWin) attention mechanism for modeling attentions in diagonal regions at hybrid scales per attention layer. The key idea of DiagSWin attention is to inject multi-scale receptive field sizes into tokens: before computing the self-attention matrix, each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity. This mechanism is able to effectively capture multi-scale context information while reducing computational complexity. With DiagSwin attention, we present a new variant of Vision Transformer models, called DiagSWin Transformers, and demonstrate their superiority in extensive experiments across various tasks. Specifically, the DiagSwin Transformer with a large size achieves 84.4% Top-1 accuracy and outperforms the SOTA CSWin Transformer on ImageNet with 40% fewer model size and computation cost. When employed as backbones, DiagSWin Transformers achieve significant improvements over the current SOTA modules. In addition, our DiagSWin-Base model yields 51.1 box mAP and 45.8 mask mAP on COCO for object detection and segmentation, and 52.3 mIoU on the ADE20K for semantic segmentation.

gSwin: Gated MLP Vision Model with Hierarchical Structure of Shifted Window

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

Gaze Estimation Based on Convolutional Structure and Sliding Window-Based Attention Mechanism

Semantic-Aware Local-Global Vision Transformer

Self-Supervised Learning with Swin Transformers

ConvMLP: Hierarchical Convolutional MLPs for Vision

Swin Transformer V2: Scaling Up Capacity and Resolution

Degenerate Swin to Win: Plain Window-based Transformer without Sophisticated Operations

DiagSWin: A multi-scale vision transformer with diagonal-shaped windows for object detection and segmentation

Vision Transformers with Hierarchical Attention

P-Swin: Parallel Swin transformer multi-scale semantic segmentation network for land cover classification

ViT-LSLA: Vision Transformer with Light Self-Limited-Attention

Robust Visual Tracking Using Hierarchical Vision Transformer with Shifted Windows Multi-Head Self-Attention

SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection

Swin-Free: Achieving Better Cross-Window Attention and Efficiency with Size-varying Window

SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentation

Hire-MLP: Vision MLP Via Hierarchical Rearrangement