Abstract:Efficient custom pooling techniques that can aggressively trim the dimensions of a feature map and thereby reduce inference compute and memory footprint for resource-constrained computer vision applications have recently gained significant traction. However, prior pooling works extract only the local context of the activation maps, limiting their effectiveness. In contrast, we propose a novel non-local self-attentive pooling method that can be used as a drop-in replacement to the standard pooling layers, such as max/average pooling or strided convolution. The proposed self-attention module uses patch embedding, multi-head self-attention, and spatial-channel restoration, followed by sigmoid activation and exponential soft-max. This self-attention mechanism efficiently aggregates dependencies between non-local activation patches during down-sampling. Extensive experiments on standard object classification and detection tasks with various convolutional neural network (CNN) architectures demonstrate the superiority of our proposed mechanism over the state-of-the-art (SOTA) pooling techniques. In particular, we surpass the test accuracy of existing pooling techniques on different variants of MobileNet-V2 on ImageNet by an average of 1.2%. With the aggressive down-sampling of the activation maps in the initial layers (providing up to 22x reduction in memory consumption), our approach achieves 1.43% higher test accuracy compared to SOTA techniques with iso-memory footprints. This enables the deployment of our models in memory-constrained devices, such as micro-controllers (without losing significant accuracy), because the initial activation maps consume a significant amount of on-chip memory for high-resolution images required for complex vision tasks. Our proposed pooling method also leverages the idea of channel pruning to further reduce memory footprints.

Delving deep into spatial pooling for squeeze-and-excitation networks

Squeeze Excitation Densely Connected Residual Convolutional Networks for Specific Emitter Identification Based on Measured Signals

Competitive Inner-Imaging Squeeze and Excitation for Residual Network

Superpixel Segmentation with Squeeze-and-excitation Networks

Squeeze-and-Excitation Wide Residual Networks in Image Classification

Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks

Aggregated Squeeze-and-excitation Transformations for Densely Connected Convolutional Networks

Self-Attentive Pooling for Efficient Deep Learning

SDE: A Novel Selective, Discriminative and Equalizing Feature Representation for Visual Recognition

Channel Locality Block: A Variant of Squeeze-and-Excitation

Semantic Segmentation Based on Spatial Pyramid Pooling and Multilayer Feature Fusion

Building Detail-Sensitive Semantic Segmentation Networks with Polynomial Pooling.

Squeeze-and-Excitation on Spatial and Temporal Deep Feature Space for Action Recognition.

Squeeze aggregated excitation network

Feature Pooling - A Feature Compression Method Used in Convolutional Neural Networks.

Enhanced Pooling Method for Convolutional Neural Networks Based on Optimal Search Theory

Gated Square-Root Pooling For Image Instance Retrieval

SPATIAL MOMENT POOLING IMPROVES NEURAL IMAGE ASSESSMENT

A2SPPNet: Attentive Atrous Spatial Pyramid Pooling Network for Salient Object Detection.

Coupled Squeeze-And-Excitation Blocks Based Cnn For Image Compression

Semantic Segmentation with Hybrid Pyramid Pooling and Stacked Pyramid Structure