Abstract:Vision transformers have shown great success on numerous computer vision tasks. However, their central component, softmax attention, prohibits vision transformers from scaling up to high-resolution images, due to both the computational complexity and memory footprint being quadratic. Linear attention was introduced in natural language processing (NLP) which reorders the self-attention mechanism to mitigate a similar issue, but directly applying existing linear attention to vision may not lead to satisfactory results. We investigate this problem and point out that existing linear attention methods ignore an inductive bias in vision tasks, i.e., 2D locality. In this paper, we propose Vicinity Attention, which is a type of linear attention that integrates 2D locality. Specifically, for each image patch, we adjust its attention weight based on its 2D Manhattan distance from its neighbouring patches. In this case, we achieve 2D locality in a linear complexity where the neighbouring image patches receive stronger attention than far away patches. In addition, we propose a novel Vicinity Attention Block that is comprised of Feature Reduction Attention (FRA) and Feature Preserving Connection (FPC) in order to address the computational bottleneck of linear attention approaches, including our Vicinity Attention, whose complexity grows quadratically with respect to the feature dimension. The Vicinity Attention Block computes attention in a compressed feature space with an extra skip connection to retrieve the original feature distribution. We experimentally validate that the block further reduces computation without degenerating the accuracy. Finally, to validate the proposed methods, we build a linear vision transformer backbone named Vicinity Vision Transformer (VVT). Targeting general vision tasks, we build VVT in a pyramid structure with progressively reduced sequence length. We perform extensive experiments on CIFAR-100, ImageNet-1 k, and ADE20 K datasets to validate the effectiveness of our method. Our method has a slower growth rate in terms of computational overhead than previous transformer-based and convolution-based networks when the input resolution increases. In particular, our approach achieves state-of-the-art image classification accuracy with 50% fewer parameters than previous approaches.

Hydra Attention: Efficient Attention with Many Heads

HydraViT: Stacking Heads for a Scalable ViT

Vision Transformers with Hierarchical Attention

Improving Vision Transformers by Overlapping Heads in Multi-Head Self-Attention

Vicinity Vision Transformer

Rethinking Attention Mechanisms in Vision Transformers with Graph Structures

Constituent Attention for Vision Transformers

Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet

You Only Need Less Attention at Each Stage in Vision Transformers

Multi-Dimensional Hyena for Spatial Inductive Bias

How Does Attention Work in Vision Transformers? A Visual Analytics Attempt

AdaViT: Adaptive Vision Transformers for Efficient Image Recognition

CAT: Cross Attention in Vision Transformer

ViT-LSLA: Vision Transformer with Light Self-Limited-Attention

LevAttention: Time, Space, and Streaming Efficient Algorithm for Heavy Attentions

Not All Images Are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length

EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention

ACC-ViT : Atrous Convolution's Comeback in Vision Transformers

FAM: Improving columnar vision transformer with feature attention mechanism

Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers