Rethinking Attention Mechanism: Channel Re-Attention and Spatial Multi-Region Attention for Fine-Grained Visual Classification

Kun Wang,Wenzong Jiang,Yanjiang Wang,Weifeng Liu,Baodi Liu

DOI: https://doi.org/10.2139/ssrn.4757424

2024-01-01

Abstract:Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Copy DOI

What problem does this paper attempt to address?

Research on classification algorithms for attention mechanism

Zhuoqun Yang,Tao Zhang,Jie Yang

DOI: https://doi.org/10.1109/DCABES50732.2020.00058

2020-01-01

Abstract:In this paper, we solve the image classification task by capturing context dependencies based on spatial and channel attention mechanism. Unlike previous research on feature fusion, we propose an attention module based on spatial and channel dimensions. This module derives attention maps respectively from spatial and channel, then for feature refinement we multiply the attention map into the feature map. Meanwhile, the module can be easily embedded into the network structures due to it is lightweight. The channel attention module selectively enhances some feature channels and suppresses certain feature channels by integrating the relationship between each feature channel. By weighting the features of all locations, spatial attention module aggregates location features. Regardless of distance, similar features are interrelated. Our module is evaluated through experiments on the ImageNet-1K and CIFAR-100 datasets.
Rethinking Attention Module Design for Point Cloud Analysis

Chengzhi Wu,Kaige Wang,Zeyun Zhong,Hao Fu,Junwei Zheng,Jiaming Zhang,Julius Pfrommer,Jürgen Beyerer

2024-07-28

Abstract:In recent years, there have been significant advancements in applying attention mechanisms to point cloud analysis. However, attention module variants featured in various research papers often operate under diverse settings and tasks, incorporating potential training strategies. This heterogeneity poses challenges in establishing a fair comparison among these attention module variants. In this paper, we address this issue by rethinking and exploring attention module design within a consistent base framework and settings. Both global-based and local-based attention methods are studied, with a focus on the selection basis and scales of neighbors for local-based attention. Different combinations of aggregated local features and computation methods for attention scores are evaluated, ranging from the initial addition/concatenation-based approach to the widely adopted dot product-based method and the recently proposed vector attention technique. Various position encoding methods are also investigated. Our extensive experimental analysis reveals that there is no universally optimal design across diverse point cloud tasks. Instead, drawing from best practices, we propose tailored attention modules for specific tasks, leading to superior performance on point cloud classification and segmentation benchmarks.

Computer Vision and Pattern Recognition
Multi-Scale Attention Constraint Network for Fine-Grained Visual Classification

Yaqing Hou,Wenkai Zhang,Dongsheng Zhou,Hongwei Ge,Qiang Zhang,Xiaopeng Wei

DOI: https://doi.org/10.1109/ICME51207.2021.9428260

2021-01-01

Abstract:Capturing subtle yet discriminative features constitutes a great challenge in fine-grained visual classification due to the large intra-class and small inter-class variances. Main-stream works for this problem localize at attention mechanism and feature relationship learning. However, existing methods treat the features in isolation while neglecting the effect of attention-enhanced features on rel...
Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions

Yichao Liu,Zongru Shao,Nico Hoffmann

DOI: https://doi.org/10.48550/arXiv.2112.05561

2021-12-10

Computer Vision and Pattern Recognition

Abstract:A variety of attention mechanisms have been studied to improve the performance of various computer vision tasks. However, the prior methods overlooked the significance of retaining the information on both channel and spatial aspects to enhance the cross-dimension interactions. Therefore, we propose a global attention mechanism that boosts the performance of deep neural networks by reducing information reduction and magnifying the global interactive representations. We introduce 3D-permutation with multilayer-perceptron for channel attention alongside a convolutional spatial attention submodule. The evaluation of the proposed mechanism for the image classification task on CIFAR-100 and ImageNet-1K indicates that our method stably outperforms several recent attention mechanisms with both ResNet and lightweight MobileNet.
Attention-based cropping and erasing learning with coarse-to-fine refinement for fine-grained visual classification

Jianpin Chen,Heng Li,Junlin Liang,Xiaofan Su,Zhenzhen Zhai,Xinyu Chai

DOI: https://doi.org/10.1016/j.neucom.2022.06.041

IF: 6

2022-01-01

Neurocomputing

Abstract:Fine-grained visual classification is challenging due to similarities within classes and discriminative features located in subtle regions. Conventional methods focus on extracting features from the most discriminative parts, which may underperform when these parts are occluded or invisible. And the limited training data also leads to serious overfitting problem. In this paper, we propose an Attentionbased Cropping and Erasing Network (ACEN) with a coarse-to-fine refinement strategy to address these problems. By convolving the feature maps from CNN, we obtain a set of attention maps which focus on discriminative object parts. Guided by the attention maps, we propose attention region cropping and erasing operations to augment training data. Moreover, the attention region cropping enhances local discriminative feature learning, and the attention region erasing promotes multi-attention learning. During inference phase, the coarse-to-fine refinement strategy is proposed to refine the model prediction. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on challenging benchmarks, including CUB-200-2011, FGVC-Aircraft and Stanford Cars. (c) 2022 Elsevier B.V. All rights reserved.
Feature Channel Adaptive Enhancement for Fine-Grained Visual Classification

Dingzhou Xie,Cheng Pang,Guanhua Wu,Rushi Lan

DOI: https://doi.org/10.1007/978-3-031-47665-5_16

2023-01-01

Abstract:Fine-grained classification poses greater challenges compared to basic-level image classification due to the visually similar sub-species. To distinguish between confusing species, we introduce a novel framework based on feature channel adaptive enhancement and attention erasure. On one hand, a lightweight module employing both channel attention and spatial attention is designed, adaptively enhancing the feature expression of important areas and obtaining more discriminative feature vectors. On the other hand, we incorporate attention erasure methods that compel the network to concentrate on less prominent areas, thereby enhancing the network’s robustness. Our method can be seamlessly integrated into various backbone networks. Finally, an evaluation of our approach is conducted across diverse public datasets, accompanied by a comprehensive comparative analysis against state-of-the-art methodologies. The experimental findings substantiate the efficacy and viability of our method in real-world scenarios, exemplifying noteworthy breakthroughs in intricate fine-grained classification endeavors.
Attention Mechanisms in Computer Vision: A Survey

Meng-Hao Guo,Tian-Xing Xu,Jiang-Jiang Liu,Zheng-Ning Liu,Peng-Tao Jiang,Tai-Jiang Mu,Song-Hai Zhang,Ralph R. Martin,Ming-Ming Cheng,Shi-Min Hu

DOI: https://doi.org/10.1007/s41095-022-0271-y

2021-11-15

Abstract:Humans can naturally and effectively find salient regions in complex scenes. Motivated by this observation, attention mechanisms were introduced into computer vision with the aim of imitating this aspect of the human visual system. Such an attention mechanism can be regarded as a dynamic weight adjustment process based on features of the input image. Attention mechanisms have achieved great success in many visual tasks, including image classification, object detection, semantic segmentation, video understanding, image generation, 3D vision, multi-modal tasks and self-supervised learning. In this survey, we provide a comprehensive review of various attention mechanisms in computer vision and categorize them according to approach, such as channel attention, spatial attention, temporal attention and branch attention; a related repository <a class="link-external link-https" href="https://github.com/MenghaoGuo/Awesome-Vision-Attentions" rel="external noopener nofollow">this https URL</a> is dedicated to collecting related work. We also suggest future directions for attention mechanism research.

Computer Vision and Pattern Recognition
Nonlocal Spatial Attention Module for Image Classification

Bingling Chen,Yan Huang,Qiaoqiao Xia,Qinglin Zhang

DOI: https://doi.org/10.1177/1729881420938927

IF: 1.714

2020-01-01

International Journal of Advanced Robotic Systems

Abstract:To enhance the capability of neural networks, research on attention mechanism have been deepened. In this area, attention modules make forward inference along channel dimension and spatial dimension sequentially, parallelly, or simultaneously. However, we have found that spatial attention modules mainly apply convolution layers to generate attention maps, which aggregate feature responses only based on local receptive fields. In this article, we take advantage of this finding to create a nonlocal spatial attention module (NL-SAM), which collects context information from all pixels to adaptively recalibrate spatial responses in a convolutional feature map. NL-SAM overcomes the limitations of repeating local operations and exports a 2D spatial attention map to emphasize or suppress responses in different locations. Experiments on three benchmark datasets show at least 0.58% improvements on variant ResNets. Furthermore, this module is simple and can be easily integrated with existing channel attention modules, such as squeeze-and-excitation and gather-excite, to exceed these significant models at a minimal additional computational cost (0.196%).
Visual Attention in Multi-Label Image Classification.

Yan Luo,Ming Jiang,Qi Zhao

DOI: https://doi.org/10.1109/cvprw.2019.00110

2019-01-01

Abstract:One of the most significant challenges in multi-label image classification is the learning of representative features that capture the rich semantic information in a cluttered scene. As an information bottleneck, the visual attention mechanism allows humans to selectively process the most important visual input, enabling rapid and accurate scene understanding. In this work, we study the correlation between visual attention and multi-label image classification, and exploit an extra attention pathway for improving multi-label image classification performance. Specifically, we propose a dual-stream neural network that consists of two sub-networks: one is a conventional classification model and the other is a saliency prediction model trained with human fixations. Features computed with the two sub-networks are trained separately and then fine-tuned jointly using a multiple cross entropy loss. Experimental results show that the additional saliency sub-network improves multi-label image classification performance on the MS COCO dataset. The improvement is consistent across various levels of scene clutterness.
Image Re-Attentionizing

Tam V. Nguyen,Bingbing Ni,Hairong Liu,Wei Xia,Jiebo Luo,Mohan Kankanhalli,Shuicheng Yan

DOI: https://doi.org/10.1109/tmm.2013.2272919

IF: 7.3

2013-01-01

IEEE Transactions on Multimedia

Abstract:In this paper, we propose a computational framework, called Image Re-Attentionizing, to endow the target region in an image with the ability of attracting human visual attention. In particular, the objective is to recolor the target patches by color transfer with naturalness and smoothness preserved yet visual attention augmented. We propose to approach this objective within the Markov Random Field (MRF) framework and an extended graph cuts method is developed to pursue the solution. The input image is first over-segmented into patches, and the patches within the target region as well as their neighbors are used to construct the consistency graphs. Within the MRF framework, the unitary potentials are defined to encourage each target patch to match the patches with similar shapes and textures from a large salient patch database, each of which corresponds to a high-saliency region in one image, while the spatial and color coherence is reinforced as pairwise potentials. We evaluate the proposed method on the direct human fixation data. The results demonstrate that the target region(s) successfully attract human attention and in the meantime both spatial and color coherence is well preserved.
Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identification

Haowei Zhu,Wenjing Ke,Dong Li,Ji Liu,Lu Tian,Yi Shan

DOI: https://doi.org/10.1109/cvpr52688.2022.00465

2022-01-01

Abstract:Recently, self-attention mechanisms have shown impressive performance in various NLP and CV tasks, which can help capture sequential characteristics and derive global information. In this work, we explore how to extend self-attention modules to better learn subtle feature embeddings for recognizing fine-grained objects, e.g., different bird species or person identities. To this end, we propose a dual cross-attention learning (DCAL) algorithm to coordinate with self-attention learning. First, we propose global-local cross-attention (GLCA) to enhance the interactions between global images and local high-response regions, which can help reinforce the spatial-wise discriminative clues for recognition. Second, we propose pairwise cross-attention (PWCA) to establish the interactions between image pairs. PWCA can regularize the attention learning of an image by treating another image as distractor and will be removed during inference. We observe that DCAL can reduce misleading attentions and diffuse the attention response to discover more complementary parts for recognition. We conduct extensive evaluations on finegrained visual categorization and object reidentification. Experiments demonstrate that DCAL performs on par with state-of-the-art methods and consistently improves multiple self-attention baselines, e.g., surpassing DeiT-Tiny and ViT-Base by 2.8% and 2.4% mAP on MSMT17, respectively.
Efficient Multi-Scale Attention Module with Cross-Spatial Learning

Daliang Ouyang,Su He,Guozhong Zhang,Mingzhu Luo,Huaiyong Guo,Jian Zhan,Zhijie Huang

DOI: https://doi.org/10.1109/ICASSP49357.2023.10096516

2023-06-06

Abstract:Remarkable effectiveness of the channel or spatial attention mechanisms for producing more discernible feature representation are illustrated in various computer vision tasks. However, modeling the cross-channel relationships with channel dimensionality reduction may bring side effect in extracting deep visual representations. In this paper, a novel efficient multi-scale attention (EMA) module is proposed. Focusing on retaining the information on per channel and decreasing the computational overhead, we reshape the partly channels into the batch dimensions and group the channel dimensions into multiple sub-features which make the spatial semantic features well-distributed inside each feature group. Specifically, apart from encoding the global information to re-calibrate the channel-wise weight in each parallel branch, the output features of the two parallel branches are further aggregated by a cross-dimension interaction for capturing pixel-level pairwise relationship. We conduct extensive ablation studies and experiments on image classification and object detection tasks with popular benchmarks (e.g., CIFAR-100, ImageNet-1k, MS COCO and VisDrone2019) for evaluating its performance.

Computer Vision and Pattern Recognition,Artificial Intelligence
Dual attention guided multi-scale CNN for fine-grained image classification

Xiaozhang Liu,Lifeng Zhang,Tao Li,Dejian Wang,Zhaojie Wang

DOI: https://doi.org/10.1016/j.ins.2021.05.040

IF: 8.1

2021-09-01

Information Sciences

Abstract:<p>For the classification of fine-grained images, the subtle differences among the subclasses of the main category must be distinguished. Intuitively, the key to realizing the fine-grained image categorization lies in locating and identifying the detailed differences in the local regions and capturing their feature representations. In this paper, we propose utilizing an attention module combined with a multi-scale latent representation network to locate the discriminative spatial regions, and then learn an accurate attention map to assist the category decision. Furthermore, an attention module is also employed to determine the channel weights of the distinct scale feature maps before the final step. Extensive experiments demonstrate that our model obtains a competitive performance against state-of-the-art baselines on two benchmark datasets, the attention validation experiments further reveal the ability of the model in choosing the proper channel features for low-quality image categorization.</p>

computer science, information systems
Focal vs Diffuse: Mechanisms of attention mediated performance enhancement in a hierarchical model of the visual system

Wang,X.,Jadi,M.

DOI: https://doi.org/10.1101/2023.05.09.540022

2023-05-10

bioRxiv

Abstract:Spatial attention is an essential cognitive process for visual perception, especially in complex scenes with poor luminance. Attentional modulation of neural activity has been documented across the visual cortex. However, how these changes in neural response lead to better behavioral performance on challenging tasks remain unknown. In our study, we implemented spatial attention in a deep convolutional neural network model of the ventral visual hierarchy and measured its impact on categorization performance on cluttered images with varying contrast levels. We applied attention to network units in three ways: enhancement in attended region (EAR), suppression in unattended region (SUAR) or both. When focally applied to a single convolutional layer, SUAR is more effective in boosting performance than EAR, especially in the presence of high contrast distractors. SUAR is also effective in recovering degraded performance due to low contrast targets, whereas EAR fails to recover. These results predict a novel mechanism of suppression of neural activity corresponding to the unattended parts of visual space. Intriguingly, EAR in our model achieves the same performance as SUAR when attention is diffusely applied to stacks of successive convolutional layers, irrespective of target contrast. This suggests an alternate mechanism of attention wherein enhancement of attended neural activity alone, when applied to successive cortical encoding stages, is an effective strategy for boosting performance in challenging object recognition tasks. Our results indicate that the two alternative attentional mechanisms are functionally equivalent in tackling challenging object recognition tasks.
Where to Focus: Investigating Hierarchical Attention Relationship for Fine-Grained Visual Classification

Yang Liu,Lei Zhou,Pengcheng Zhang,Xiao Bai,Lin Gu,Xiaohan Yu,Jun Zhou,Edwin R. Hancock

DOI: https://doi.org/10.1007/978-3-031-20053-3_4

2022-01-01

Abstract:Object categories are often grouped into a multi-granularity taxonomic hierarchy. Classifying objects at coarser-grained hierarchy requires global and common characteristics, while finer-grained hierarchy classification relies on local and discriminative features. Therefore, humans should also subconsciously focus on different object regions when classifying different hierarchies. This granularity-wise attention is confirmed by our collected human real-time gaze data on different hierarchy classifications. To leverage this mechanism, we propose a CrossHierarchical Region Feature (CHRF) learning framework. Specifically, we first design a region feature mining module that imitates humans to learn different granularity-wise attention regions with multi-grained classification tasks. To explore how human attention shifts from one hierarchy to another, we further present a cross-hierarchical orthogonal fusion module to enhance the region feature representation by blending the original feature and an orthogonal component extracted from adjacent hierarchies. Experiments on five hierarchical fine-grained datasets demonstrate the effectiveness of CHRF compared with the state-of-the-art methods. Ablation study and visualization results also consistently verify the advantages of our human attention-oriented modules. The code and dataset are available at https://github.com/visiondom/CHRF.
Discriminative Region Enhancing and Suppression Network for Fine-Grained Visual Categorization

Guanhua Wu,Cheng Pang,Rushi Lan,Yilin Zhang,Pingping Zhou

DOI: https://doi.org/10.1007/978-3-031-47665-5_8

2023-01-01

Abstract:Attention mechanisms are intensively devoted to local feature abstraction for fine-grained visual categorization. A limitation of attention-based methods is that they focus on salient region mining and feature extraction, while ignoring the ability to incorporate discriminative and complementary features from other parts of the image. In order to address this issue, we introduce a novel network known as the Discriminative Region Enhancing and Suppression Network (DRESNet). This network efficiently extracts a wide range of diverse and complementary features, thereby enhancing the final representation. Specifically, a plug-and-play salient region diffusion (SRD) module is proposed to explicitly enhances the salient features extracted by any backbone network. The SRD module can adaptively adjust the weights of regions and redirect attention to other non-discriminative regions to generate different complementary features. The proposed discriminative region enhancing and suppression network is free from bounding boxes or part annotations and can be trained end-to-end. Our proposed method demonstrates competitive performance on three fine-grained classification benchmark datasets, as supported by extensive experimental results. Additionally, it is compatible with widely used frameworks currently in use.
Attention Graph: Learning Effective Visual Features for Large-Scale Image Classification

Xuelian Cui,Zhanjie Zhang,Tao Zhang,Zhuoqun Yang,Jie Yang

DOI: https://doi.org/10.1177/17483026211065375

2022-01-01

Journal of Algorithms & Computational Technology

Abstract:In recent years, the research of deep learning has received extensive attention, and many breakthroughs have been made in various fields. On this basis, a neural network with the attention mechanism has become a research hotspot. In this paper, we try to solve the image classification task by implementing channel and spatial attention mechanism which improve the expression ability of neural network model. Different from previous studies, we propose an attention module consisting of channel attention module (CAM) and spatial attention module (SAM). The proposed module derives attention graphs from channel dimension and spatial dimension respectively, then the input features are selectively learned according to the importance of the features. Besides, this module is lightweight and can be easily integrated into image classification algorithms. In the experiment, we combine the deep residual network model with the attention module and the experimental results show that the proposed method brings higher image classification accuracy. The channel attention module adds weight to the signals on different convolution channels to represent the correlation. For different channels, the higher the weight, the higher the correlation which required more attention. The main function of spatial attention is to capture the most informative part in the local feature graph, which is a supplement to channel attention. We evaluate our proposed module based on the ImageNet-1K and Cifar-100 respectively. Through a large number of comparative experiments, our proposed model achieved outstanding performance.
Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification

Yongming Rao,Guangyi Chen,Jiwen Lu,Jie Zhou

DOI: https://doi.org/10.1109/iccv48922.2021.00106

2021-01-01

Abstract:Attention mechanism has demonstrated great potential in fine-grained visual recognition tasks. In this paper, we present a counterfactual attention learning method to learn more effective attention based on causal inference. Unlike most existing methods that learn visual attention based on conventional likelihood, we propose to learn the attention with counterfactual causality, which provides a tool to measure the attention quality and a powerful supervisory signal to guide the learning process. Specifically, we analyze the effect of the learned visual attention on network prediction through counterfactual intervention and maximize the effect to encourage the network to learn more useful attention for fine-grained image recognition. Empirically, we evaluate our method on a wide range of fine-grained recognition tasks where attention plays a crucial role, including fine-grained image categorization, person re-identification, and vehicle re-identification. The consistent improvement on all benchmarks demonstrates the effectiveness of our method. (1)
Channelized Axial Attention for Semantic Segmentation -- Considering Channel Relation within Spatial Attention for Semantic Segmentation

Ye Huang,Di Kang,Wenjing Jia,Xiangjian He,Liu Liu

DOI: https://doi.org/10.48550/arXiv.2101.07434

2021-09-12

Abstract:Spatial and channel attentions, modelling the semantic interdependencies in spatial and channel dimensions respectively, have recently been widely used for semantic segmentation. However, computing spatial and channel attentions separately sometimes causes errors, especially for those difficult cases. In this paper, we propose Channelized Axial Attention (CAA) to seamlessly integrate channel attention and spatial attention into a single operation with negligible computation overhead. Specifically, we break down the dot-product operation of the spatial attention into two parts and insert channel relation in between, allowing for independently optimized channel attention on each spatial location. We further develop grouped vectorization, which allows our model to run with very little memory consumption without slowing down the running speed. Comparative experiments conducted on multiple benchmark datasets, including Cityscapes, PASCAL Context, and COCO-Stuff, demonstrate that our CAA outperforms many state-of-the-art segmentation models (including dual attention) on all tested datasets.

Computer Vision and Pattern Recognition

Rethinking Attention Mechanism: Channel Re-Attention and Spatial Multi-Region Attention for Fine-Grained Visual Classification

Research on classification algorithms for attention mechanism

Rethinking Attention Module Design for Point Cloud Analysis

Multi-Scale Attention Constraint Network for Fine-Grained Visual Classification

Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions

Attention-based cropping and erasing learning with coarse-to-fine refinement for fine-grained visual classification

Feature Channel Adaptive Enhancement for Fine-Grained Visual Classification

Attention Mechanisms in Computer Vision: A Survey

Nonlocal Spatial Attention Module for Image Classification

Visual Attention in Multi-Label Image Classification.

Image Re-Attentionizing

Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identification

Efficient Multi-Scale Attention Module with Cross-Spatial Learning

Dual attention guided multi-scale CNN for fine-grained image classification

Focal vs Diffuse: Mechanisms of attention mediated performance enhancement in a hierarchical model of the visual system

Where to Focus: Investigating Hierarchical Attention Relationship for Fine-Grained Visual Classification

Discriminative Region Enhancing and Suppression Network for Fine-Grained Visual Categorization

Attention Graph: Learning Effective Visual Features for Large-Scale Image Classification

Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification

Channelized Axial Attention for Semantic Segmentation -- Considering Channel Relation within Spatial Attention for Semantic Segmentation