Abstract:Self-attention mechanism has been a key factor in the recent progress of Vision Transformer (ViT), which enables adaptive feature extraction from global contexts. However, existing self-attention methods either adopt sparse global attention or window attention to reduce the computation complexity, which may compromise the local feature learning or subject to some handcrafted designs. In contrast, local attention, which restricts the receptive field of each query to its own neighboring pixels, enjoys the benefits of both convolution and self-attention, namely local inductive bias and dynamic feature selection. Nevertheless, current local attention modules either use inefficient Im2Col function or rely on specific CUDA kernels that are hard to generalize to devices without CUDA support. In this paper, we propose a novel local attention module, Slide Attention, which leverages common convolution operations to achieve high efficiency, flexibility and generalizability. Specifically, we first re-interpret the column-based Im2Col function from a new row-based perspective and use Depthwise Convolution as an efficient substitution. On this basis, we propose a deformed shifting module based on the re-parameterization technique, which further relaxes the fixed key/value positions to deformed features in the local region. In this way, our module realizes the local attention paradigm in both efficient and flexible manner. Extensive experiments show that our slide attention module is applicable to a variety of advanced Vision Transformer models and compatible with various hardware devices, and achieves consistently improved performances on comprehensive benchmarks. Code is available at <a class="link-external link-https" href="https://github.com/LeapLabTHU/Slide-Transformer" rel="external noopener nofollow">this https URL</a>.

A Window Attention Based Transformer for Automatic Speech Recognition

LAS-Transformer: An Enhanced Transformer Based on the Local Attention Mechanism for Speech Recognition

Research Status and Prospect of Transformer in Speech Recognition

Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition

Beyond Fixation: Dynamic Window Visual Transformer

WA-Transformer: Window Attention-based Transformer with Two-stage Strategy for Multi-task Audio Source Separation

Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

End-to-End ASR with Adaptive Span Self-Attention

DWFormer: Dynamic Window transFormer for Speech Emotion Recognition

DSWA: A Dilated Shift Window Attention Method for Chinese Named Entity Recognition (S).

SIMPLIFIED SELF-ATTENTION FOR TRANSFORMER-BASED END-TO-END SPEECH RECOGNITION

Enhanced Window-Based Self-Attention with Global and Multi-Scale Representations for Remote Sensing Image Super-Resolution

Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers

Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism

Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognition

The Speechtransformer for Large-scale Mandarin Chinese Speech Recognition.

Speech Keyword Spotting Method Based on Swin-Transformer Model

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Explainability of Speech Recognition Transformers Via Gradient-Based Attention Visualization

VSA: Learning Varied-Size Window Attention in Vision Transformers