Abstract:Self-attention mechanism has been a key factor in the recent progress of Vision Transformer (ViT), which enables adaptive feature extraction from global contexts. However, existing self-attention methods either adopt sparse global attention or window attention to reduce the computation complexity, which may compromise the local feature learning or subject to some handcrafted designs. In contrast, local attention, which restricts the receptive field of each query to its own neighboring pixels, enjoys the benefits of both convolution and self-attention, namely local inductive bias and dynamic feature selection. Nevertheless, current local attention modules either use inefficient Im2Col function or rely on specific CUDA kernels that are hard to generalize to devices without CUDA support. In this paper, we propose a novel local attention module, Slide Attention, which leverages common convolution operations to achieve high efficiency, flexibility and generalizability. Specifically, we first re-interpret the column-based Im2Col function from a new row-based perspective and use Depthwise Convolution as an efficient substitution. On this basis, we propose a deformed shifting module based on the re-parameterization technique, which further relaxes the fixed key/value positions to deformed features in the local region. In this way, our module realizes the local attention paradigm in both efficient and flexible manner. Extensive experiments show that our slide attention module is applicable to a variety of advanced Vision Transformer models and compatible with various hardware devices, and achieves consistently improved performances on comprehensive benchmarks. Code is available at <a class="link-external link-https" href="https://github.com/LeapLabTHU/Slide-Transformer" rel="external noopener nofollow">this https URL</a>.

Local or Global? A Novel Transformer for Chinese Named Entity Recognition Based on Multi-View and Sliding Attention

Local Slot Attention for Vision-and-Language Navigation

A Local Information Perception Enhancement–Based Method for Chinese NER

Chinese Medical Named Entity Recognition Based on Fusion of Global Features and Multi-Local Features

Branch-Transformer: A Parallel Branch Architecture to Capture Local and Global Features for Language Identification

Local-to-Global Self-Attention in Vision Transformers

MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition

A hybrid Transformer approach for Chinese NER with features augmentation

Augmented Transformers with Adaptive n-grams Embedding for Multilingual Scene Text Recognition

Scene Chinese Recognition with Local and Global Attention

HA-Transformer: Harmonious aggregation from local to global for object detection

Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

AttentionViz: A Global View of Transformer Attention

Multi-task Transformer with Relation-attention and Type-attention for Named Entity Recognition

Modeling Local Dependence in Natural Language with Multi-channel Recurrent Neural Networks

An Improved Chinese Named Entity Recognition Method with TB-LSTM-CRF

Hero-Gang Neural Model For Named Entity Recognition

Unifying Global-Local Representations in Salient Object Detection with Transformer

Local Information Modeling with Self-Attention for Speaker Verification

TENER: Adapting Transformer Encoder for Name Entity Recognition