Abstract:Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such a fine-grained level. In fact, this is the root cause of the scalability issue w.r.t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel \textbf{at different granularity levels} instead. Specifically, in each layer, we use two different ways to represent an image: a fine-grained regular grid and a coarse-grained set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named \textit{GLMix}: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e.g., 64) semantic slots to match the performance of recent state-of-the-art backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces a meaningful semantic grouping effect with only IN1k classification supervision, which may induce better interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available at \url{<a class="link-external link-https" href="https://github.com/rayleizhu/GLMix" rel="external noopener nofollow">this https URL</a>}.

HMeanMax: Placing HMAX and HoG into a unified framework

Hmax Model: A Survey

Hamming Embedding Sensitivity Guided Fusion Network for 3D Shape Representation.

Contextual Max Pooling for Human Action Recognition

Distributed Sparse Hmax Model

C-HMAX: Artificial Cognitive Model Inspired by the Color Vision Mechanism of the Human Brain

An Unsupervised Spiking Deep Neural Network For Object Recognition

Sparsity-Regularized Hmax For Visual Recognition

A Novel Bio-inspired Image Recognition Network with Extreme Learning Machine

A HMAX with LLC for visual recognition

MAXIM: Multi-Axis MLP for Image Processing

Bio-Inspired Object Recognition Using Parameterized Metric Learning.

Adaptive Facial Expression Recognition Method Based on MBP and HMOG Feature

Deep Generalized Max Pooling

SISCHMAX: Discovering Common Contour Patterns

Deep hybrid manifold for image set classification

MultiMax: Sparse and Multi-Modal Attention Learning

DHM-Net: Deep Hypergraph Modeling for Robust Feature Matching

Spatial pooling of heterogeneous features for image applications.

Feature Learning in Image Hierarchies using Functional Maximal Correlation

Revisiting the Integration of Convolution and Attention for Vision Backbone