Abstract:Global pooling is one of the most significant operations in many machine learning models and tasks, which works for information fusion and structured data (like sets and graphs) representation. However, without solid mathematical fundamentals, its practical implementations often depend on empirical mechanisms and thus lead to sub-optimal, even unsatisfactory performance. In this work, we develop a novel and generalized global pooling framework through the lens of optimal transport. The proposed framework is interpretable from the perspective of expectation-maximization. Essentially, it aims at learning an optimal transport across sample indices and feature dimensions, making the corresponding pooling operation maximize the conditional expectation of input data. We demonstrate that most existing pooling methods are equivalent to solving a regularized optimal transport (ROT) problem with different specializations, and more sophisticated pooling operations can be implemented by hierarchically solving multiple ROT problems. Making the parameters of the ROT problem learnable, we develop a family of regularized optimal transport pooling (ROTP) layers. We implement the ROTP layers as a new kind of deep implicit layer. Their model architectures correspond to different optimization algorithms. We test our ROTP layers in several representative set-level machine learning scenarios, including multi-instance learning (MIL), graph classification, graph set representation, and image classification. Experimental results show that applying our ROTP layers can reduce the difficulty of the design and selection of global pooling -- our ROTP layers may either imitate some existing global pooling methods or lead to some new pooling layers fitting data better. The code is available at \url{<a class="link-external link-https" href="https://github.com/SDS-Lab/ROT-Pooling" rel="external noopener nofollow">this https URL</a>}.

R2FP: Rich and Robust Feature Pooling for Mining Visual Data

Combining Local and Global: Rich and Robust Feature Pooling for Visual Recognition.

Learning Relative Features Through Adaptive Pooling for Image Classification

Spatial pooling of heterogeneous features for image applications.

Adaptive Bilinear Pooling for Fine-grained Representation Learning.

Scene Aligned Pooling For Complex Video Recognition

Long Range Pooling for 3D Large-Scale Scene Understanding

Deep Regional Feature Pooling for Video Matching

Face Image Classification by Pooling Raw Features

Image Classification by Hierarchical Spatial Pooling with Partial Least Squares Analysis.

Information Entropy Based Feature Pooling for Convolutional Neural Networks

Stochastic Region Pooling: Make Attention More Expressive

Unsupervised Feature Learning For Rgb-D Image Classification

Regularized Optimal Transport Layers for Generalized Global Pooling Operations

Weakly supervised sparse coding with geometric consistency pooling

Rethinking Pooling for Multi-Granularity Features in Aerial-View Geo-Localization

Simultaneous Spectral-Spatial Feature Selection and Extraction for Hyperspectral Images

A Pooling-Based Feature Pyramid Network for Salient Object Detection

Shared Coupling-bridge for Weakly Supervised Local Feature Learning

Self-Attentive Pooling for Efficient Deep Learning

Robust Multiview Feature Learning for RGB-D Image Understanding