Abstract:Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information, amplifying critical details to focus more on the essential aspects of data. Deep learning has employed attention to boost performance for many applications. Interestingly, the same attention design can suit processing different data modalities and can easily be incorporated into large networks. Furthermore, multiple complementary attention mechanisms can be incorporated into one network. Hence, attention techniques have become extremely attractive. However, the literature lacks a comprehensive survey on attention techniques to guide researchers in employing attention in their deep models. Note that, besides being demanding in terms of training data and computational resources, transformers only cover a single category in self-attention out of the many categories available. We fill this gap and provide an in-depth survey of 50 attention techniques, categorizing them by their most prominent features. We initiate our discussion by introducing the fundamental concepts behind the success of the attention mechanism. Next, we furnish some essentials such as the strengths and limitations of each attention category, describe their fundamental building blocks, basic formulations with primary usage, and applications specifically for computer vision. We also discuss the challenges and general open questions related to attention mechanisms. Finally, we recommend possible future research directions for deep attention. All the information about visual attention methods in deep learning is provided at \href{<a class="link-external link-https" href="https://github.com/saeed-anwar/VisualAttention" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/saeed-anwar/VisualAttention" rel="external noopener nofollow">this https URL</a>}

Attention-based CNNs for Image Classification: A Survey

HAM: Hybrid Attention Module in Deep Convolutional Neural Networks for Image Classification

Attention mechanisms in computer vision: A survey

Attention Graph: Learning Effective Visual Features for Large-Scale Image Classification

An Overview of the Attention Mechanisms in Computer Vision

Dense Attention Convolutional Network for Image Classification

Research on classification algorithms for attention mechanism

Hyperspectral Image Classification with Attention Aided CNNs

Visual Attention Methods in Deep Learning: An In-Depth Survey

Attention Models for Point Clouds in Deep Learning: A Survey

Improved deep CNNs based on Nonlinear Hybrid Attention Module for image classification

Mixed Attention Mechanism for Small-Sample Fine-grained Image Classification

A convolutional attention model for text classification

Nonlocal Spatial Attention Module for Image Classification

The Application of Two-Level Attention Models in Deep Convolutional Neural Network for Fine-Grained Image Classification

Class attention network for image recognition

Attention, please! A survey of neural attention models in deep learning

A convolutional attentional neural network for sentiment classification

Densely Connected CNN with Multi-scale Feature Attention for Text Classification

Non-Local Attention Learning for Medical Image Classification