Abstract:Attention mechanisms have gradually become necessary to enhance the representational power of convolutional neural networks (CNNs). Despite recent progress in attention mechanism research, some open problems still exist. Most existing methods ignore modeling multi-scale feature representations, structural information, and long-range channel dependencies, which are essential for delivering more discriminative attention maps. This study proposes a novel, low-overhead, high-performance attention mechanism with strong generalization ability for various networks and datasets. This mechanism is called Multi-Scale Spatial Pyramid Attention (MSPA) and can be used to solve the limitations of other attention methods. For the critical components of MSPA, we not only develop the Hierarchical-Phantom Convolution (HPC) module, which can extract multi-scale spatial information at a more granular level utilizing hierarchical residual-like connections, but also design the Spatial Pyramid Recalibration (SPR) module, which can integrate structural regularization and structural information in an adaptive combination mechanism, while employing the Softmax operation to build long-range channel dependencies. The proposed MSPA is a powerful tool that can be conveniently embedded into various CNNs as a plug-and-play component. Correspondingly, using MSPA to replace the 3 × 3 convolution in the bottleneck residual blocks of ResNets, we created a series of simple and efficient backbones named MSPANet, which naturally inherit the advantages of MSPA. Without bells and whistles, our method substantially outperforms other state-of-the-art counterparts in all evaluation metrics based on extensive experimental results from CIFAR-100 and ImageNet-1K image recognition. When applying MSPA to ResNet-50, our model achieves top-1 classification accuracy of 81.74% and 78.40% on the CIFAR-100 and ImageNet-1K benchmarks, exceeding the corresponding baselines by 3.95% and 2.27%, respectively. We also obtained promising performance improvements of 1.15% and 0.91% compared to the competitive EPSANet-50. In addition, empirical research results in autonomous driving engineering applications also demonstrate that our method can significantly improve the accuracy and real-time performance of image recognition with cheaper overhead. Our code is publicly available at https://github.com/ndsclark/MSPANet .

Cascaded Multi-Scale Attention for Enhanced Multi-Scale Feature Extraction and Interaction with Low-Resolution Images

Multi-Scale Cross-Attention Fusion Network Based on Image Super-Resolution

A Cnn-Based Depth Estimation Approach With Multi-Scale Sub-Pixel Convolutions And A Smoothness Constraint

MCA: Multidimensional Collaborative Attention in Deep Convolutional Neural Networks for Image Recognition

Super-Resolution for Monocular Depth Estimation with Multi-Scale Sub-Pixel Convolutions and a Smoothness Constraint.

Multi-scale spatial pyramid attention mechanism for image recognition: An effective approach

Multi-Scale Cross-Modal Spatial Attention Fusion for Multi-label Image Recognition.

Multi-scale coupled attention for visual object detection

Multi-scale attention network for image super-resolution

A Multiscale Attention Network for Remote Sensing Scene Images Classification

MCANet: Medical Image Segmentation with Multi-Scale Cross-Axis Attention

Multi-Scale Spatial and Channel-wise Attention for Improving Object Detection in Remote Sensing Imagery

A Multi-Scale Layer-Channel Attention Network for Image Super-Resolution

Multi-Scale Context Attention Network for Image Retrieval.

Efficient Multi-Scale Attention Module with Cross-Spatial Learning

Multi-Scale Non-Local Sparse Attention for Single Image Super-Resolution.

Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition

A channel-wise multi-scale network for single image super-resolution

A Multi-scale Contextual Attention Mechanism for Convolutional Neural Networks

Multibranch Spatial-Channel Attention for Semantic Labeling of Very High-Resolution Remote Sensing Images.

MSMHSA-DeepLab V3+: An Effective Multi-Scale, Multi-Head Self-Attention Network for Dual-Modality Cardiac Medical Image Segmentation