Abstract:Efficient custom pooling techniques that can aggressively trim the dimensions of a feature map and thereby reduce inference compute and memory footprint for resource-constrained computer vision applications have recently gained significant traction. However, prior pooling works extract only the local context of the activation maps, limiting their effectiveness. In contrast, we propose a novel non-local self-attentive pooling method that can be used as a drop-in replacement to the standard pooling layers, such as max/average pooling or strided convolution. The proposed self-attention module uses patch embedding, multi-head self-attention, and spatial-channel restoration, followed by sigmoid activation and exponential soft-max. This self-attention mechanism efficiently aggregates dependencies between non-local activation patches during down-sampling. Extensive experiments on standard object classification and detection tasks with various convolutional neural network (CNN) architectures demonstrate the superiority of our proposed mechanism over the state-of-the-art (SOTA) pooling techniques. In particular, we surpass the test accuracy of existing pooling techniques on different variants of MobileNet-V2 on ImageNet by an average of 1.2%. With the aggressive down-sampling of the activation maps in the initial layers (providing up to 22x reduction in memory consumption), our approach achieves 1.43% higher test accuracy compared to SOTA techniques with iso-memory footprints. This enables the deployment of our models in memory-constrained devices, such as micro-controllers (without losing significant accuracy), because the initial activation maps consume a significant amount of on-chip memory for high-resolution images required for complex vision tasks. Our proposed pooling method also leverages the idea of channel pruning to further reduce memory footprints.

Ask the Dictionary: Soft-Assignment Location-Orientation Pooling for Image Classification

Image Classification by Hierarchical Spatial Pooling with Partial Least Squares Analysis.

Weakly supervised sparse coding with geometric consistency pooling

Considering The Spatial Layout Information Of Bag Of Features (Bof) Framework For Image Classification

Boat in the Sky: Background Decoupling and Object-aware Pooling for Weakly Supervised Semantic Segmentation

Generalized regular spatial pooling for image classification

Learning Class-Specific Pooling Shapes for Image Classification.

Weakly-Supervised Semantic Segmentation with Visual Words Learning and Hybrid Pooling

Wasserstein Pooling for Image Classification

BoW Pooling: A Plug-and-Play Unit for Feature Aggregation of Point Clouds

Adjacent Coding for Image Classification.

Spatial pooling of heterogeneous features for image applications.

SFCM: Learn a Pooling Kernel for Weakly Supervised Object Localization

Self-Attentive Pooling for Efficient Deep Learning

Gradient Corner Pooling for Keypoint-Based Object Detection

Learning attribute-aware dictionary for image classification and search

CSPS: An Adaptive Pooling Method for Image Classification

Spatial pooling of heterogeneous features for image classification.

Simple Techniques Make Sense: Feature Pooling and Normalization for Image Classification

Learning Relative Features Through Adaptive Pooling for Image Classification

SoftPoolNet: Shape Descriptor for Point Cloud Completion and Classification