Abstract:Efficient custom pooling techniques that can aggressively trim the dimensions of a feature map and thereby reduce inference compute and memory footprint for resource-constrained computer vision applications have recently gained significant traction. However, prior pooling works extract only the local context of the activation maps, limiting their effectiveness. In contrast, we propose a novel non-local self-attentive pooling method that can be used as a drop-in replacement to the standard pooling layers, such as max/average pooling or strided convolution. The proposed self-attention module uses patch embedding, multi-head self-attention, and spatial-channel restoration, followed by sigmoid activation and exponential soft-max. This self-attention mechanism efficiently aggregates dependencies between non-local activation patches during down-sampling. Extensive experiments on standard object classification and detection tasks with various convolutional neural network (CNN) architectures demonstrate the superiority of our proposed mechanism over the state-of-the-art (SOTA) pooling techniques. In particular, we surpass the test accuracy of existing pooling techniques on different variants of MobileNet-V2 on ImageNet by an average of 1.2%. With the aggressive down-sampling of the activation maps in the initial layers (providing up to 22x reduction in memory consumption), our approach achieves 1.43% higher test accuracy compared to SOTA techniques with iso-memory footprints. This enables the deployment of our models in memory-constrained devices, such as micro-controllers (without losing significant accuracy), because the initial activation maps consume a significant amount of on-chip memory for high-resolution images required for complex vision tasks. Our proposed pooling method also leverages the idea of channel pruning to further reduce memory footprints.

Two-stage Pooling of Deep Convolutional Features for Image Retrieval.

Exploring Geometric Information in CNN for Image Retrieval.

Aggregating Hierarchical Binary Activations for Image Retrieval

Exploiting Hierarchical Activations of Neural Network for Image Retrieval.

Learning Feature Embedding with Strong Neural Activations for Fine-Grained Retrieval

Selective Convolutional Features based Generalized-mean Pooling for Fine-grained Image Retrieval

Gated Square-Root Pooling For Image Instance Retrieval

Image retrieval based on aggregated deep features weighted by regional significance and channel sensitivity

Cross-convolutional-layer Pooling for Generic Visual Recognition.

Multi-scale Orderless Cross-Regions-pooling of Deep Attributes for Image Retrieval

Where to Focus: Query Adaptive Matching for Instance Retrieval Using Convolutional Feature Maps.

Object Level Deep Feature Pooling for Compact Image Representation

Multi-level Image Representation for Large-Scale Image-Based Instance Retrieval.

Self-Attentive Pooling for Efficient Deep Learning

Unsupervised Semantic-Based Convolutional Features Aggregation for Image Retrieval

REMAP: Multi-Layer Entropy-Guided Pooling of Dense CNN Features for Image Retrieval

Deep Aggregation of Regional Convolutional Activations for Content Based Image Retrieval

Exploiting the Complementary Strengths of Multi-Layer CNN Features for Image Retrieval.

Kernelized Subspace Pooling for Deep Local Descriptors.

Subspace-based Multi-View Fusion for Instance-Level Image Retrieval

Combining Self-Aggregation Vector and Indicator Matrix for Improved Object Retrieval