Abstract:Efficient custom pooling techniques that can aggressively trim the dimensions of a feature map and thereby reduce inference compute and memory footprint for resource-constrained computer vision applications have recently gained significant traction. However, prior pooling works extract only the local context of the activation maps, limiting their effectiveness. In contrast, we propose a novel non-local self-attentive pooling method that can be used as a drop-in replacement to the standard pooling layers, such as max/average pooling or strided convolution. The proposed self-attention module uses patch embedding, multi-head self-attention, and spatial-channel restoration, followed by sigmoid activation and exponential soft-max. This self-attention mechanism efficiently aggregates dependencies between non-local activation patches during down-sampling. Extensive experiments on standard object classification and detection tasks with various convolutional neural network (CNN) architectures demonstrate the superiority of our proposed mechanism over the state-of-the-art (SOTA) pooling techniques. In particular, we surpass the test accuracy of existing pooling techniques on different variants of MobileNet-V2 on ImageNet by an average of 1.2%. With the aggressive down-sampling of the activation maps in the initial layers (providing up to 22x reduction in memory consumption), our approach achieves 1.43% higher test accuracy compared to SOTA techniques with iso-memory footprints. This enables the deployment of our models in memory-constrained devices, such as micro-controllers (without losing significant accuracy), because the initial activation maps consume a significant amount of on-chip memory for high-resolution images required for complex vision tasks. Our proposed pooling method also leverages the idea of channel pruning to further reduce memory footprints.

Detachable Second-Order Pooling: Toward High-Performance First-Order Networks

Second-Order Pooling for Graph Neural Networks

Face Identification with Second-Order Pooling in Single-Layer Networks

Cross-convolutional-layer Pooling for Generic Visual Recognition.

Face Identification with Second-Order Pooling

First and Second-Order Information Fusion Networks for Remote Sensing Scene Classification

Self-Attentive Pooling for Efficient Deep Learning

Attention-Based Second-Order Pooling Network for Hyperspectral Image Classification

Few-Shot Object Detection by Second-Order Pooling

PCANet-II: when PCANet Meets the Second Order Pooling.

Is Second-order Information Helpful for Large-scale Visual Recognition?

LIP: Local Importance-Based Pooling

Detail-Preserving Pooling in Deep Networks

Two-stage Pooling of Deep Convolutional Features for Image Retrieval.

RunPool: A Dynamic Pooling Layer for Convolution Neural Network

Equivariant vs. Invariant Layers: A Comparison of Backbone and Pooling for Point Cloud Classification

Enhanced mechanisms of pooling and channel attention for deep learning feature maps

Multi Layer Neural Networks as Replacement for Pooling Operations

Second-Order Attention Network For Single Image Super-Resolution

Analysing the Effects of Pooling Combinations on Invariance to Position and Deformation in Convolutional Neural Networks

TI-POOLING: transformation-invariant pooling for feature learning in Convolutional Neural Networks