Abstract:This paper tackles the high computational/space complexity associated with Multi-Head Self-Attention (MHSA) in vanilla vision transformers. To this end, we propose Hierarchical MHSA (H-MHSA), a novel approach that computes self-attention in a hierarchical fashion. Specifically, we first divide the input image into patches as commonly done, and each patch is viewed as a token. Then, the proposed H-MHSA learns token relationships within local patches, serving as local relationship modeling. Then, the small patches are merged into larger ones, and H-MHSA models the global dependencies for the small number of the merged tokens. At last, the local and global attentive features are aggregated to obtain features with powerful representation capacity. Since we only calculate attention for a limited number of tokens at each step, the computational load is reduced dramatically. Hence, H-MHSA can efficiently model global relationships among tokens without sacrificing fine-grained information. With the H-MHSA module incorporated, we build a family of Hierarchical-Attention-based Transformer Networks, namely HAT-Net. To demonstrate the superiority of HAT-Net in scene understanding, we conduct extensive experiments on fundamental vision tasks, including image classification, semantic segmentation, object detection, and instance segmentation. Therefore, HAT-Net provides a new perspective for vision transformers. Code and pretrained models are available at <a class="link-external link-https" href="https://github.com/yun-liu/HAT-Net" rel="external noopener nofollow">this https URL</a>.

A Multiscale Visualization of Attention in the Transformer Model

A Multiscale Visualization of Attention in the Transformer Model

AttentionViz: A Global View of Transformer Attention

Constituent Attention for Vision Transformers

How Does Attention Work in Vision Transformers? A Visual Analytics Attempt

Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models

Multi-manifold Attention for Vision Transformers

Dodrio: Exploring Transformer Models with Interactive Visualization

Vision Transformers with Hierarchical Attention

VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers

Unveiling and Controlling Anomalous Attention Distribution in Transformers

Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

Self-Attention in Transformer Networks Explains Monkeys' Gaze Pattern in Pac-Man Game

Improving Vision Transformers by Overlapping Heads in Multi-Head Self-Attention

CAT: Cross Attention in Vision Transformer

Context-Sensitive Visualization of Deep Learning Natural Language Processing Models

FAM: Improving columnar vision transformer with feature attention mechanism

A Primal-Dual Framework for Transformers and Neural Networks

Predictive Attention Transformer: Improving Transformer with Attention Map Prediction