Abstract:There is an inherent and longstanding challenge for vision learners to exploit informative features from digital images with spatial redundancy. Given pre-processing image methods require task-specific customization and may rise unanticipated poor performance due to redundancy removal, we explore improving vision learners to combat spatial redundancy during vision learning, a task-agnostic and robust solution. Among popular vision learners, vision transformers with self-attention can mitigate pixel redundancy by capturing global dependencies, while convolutional learners fall into locality via a limited receptive field. To this end, based on investigating inter-pixel spatial redundancy of images, in this work, we propose spectral norm attention (SNA), a novel yet efficient attention block to help convolutional neural networks (CNNs) highlight informative features. We can seamlessly plug SNA into off-the-shelf CNNs to suppress the contributions of redundant features by globally differentiating and weighting. In particular, SNA performs singular value decomposition (SVD) on intermediate features of each image within a mini-batch to obtain its spectral norm. The features in the direction of the spectral norm are most informative, while the discriminative features in other directions leave less. Hence, we apply the rank-one approximation of the spectral norm direction as attention weights to enhance informative features. Besides, we adopt the power iteration algorithm to approximate the spectral norm to significantly reduce the matrix computation overhead during training, thus keeping inference speed on par with vanilla CNNs. We extensively evaluate our SNA on four mainstream natural datasets to demonstrate the effectiveness and favourability of our SNA against its counterparts. In addition, the experimental results of image classification and object detection show our SNA can bring more gains to medical images with heavy redundancy than other state-of-the-art attention modules.

Double Spatial Attention for Convolution Neural Networks

HAM: Hybrid Attention Module in Deep Convolutional Neural Networks for Image Classification

DAS: A Deformable Attention to Capture Salient Information in CNNs

A Symmetric Efficient Spatial and Channel Attention (ESCA) Module Based on Convolutional Neural Networks

CSA-Net: Channel-wise Spatially Autocorrelated Attention Networks

DSAT-Net: Dual Spatial Attention Transformer for Building Extraction From Aerial Images

ELA: Efficient Local Attention for Deep Convolutional Neural Networks

Combating spatial redundancy with spectral norm attention in convolutional learners

An Attention Module for Convolutional Neural Networks

A feature-wise attention module based on the difference with surrounding features for convolutional neural networks

Spatial and Channel Dimensions Attention Feature Transfer for Better Convolutional Neural Networks

Efficient Multi-Scale Attention Module with Cross-Spatial Learning

PCSA: Enhancing CNN Performance With Pyramid Channel and Spatial Attention

Double Attention Convolutional Neural Network for Driver Action Recognition

A Spatial–Channel–Temporal-Fused Attention for Spiking Neural Networks

A Novel Global Spatial Attention Mechanism in Convolutional Neural Network for Medical Image Classification

A Multi-scale Contextual Attention Mechanism for Convolutional Neural Networks

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

Joint Spatial and Layer Attention for Convolutional Networks

Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions

ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks