Abstract:Recent studies have integrated convolution into transformers to introduce inductive bias and improve generalization performance. However, the static nature of conventional convolution prevents it from dynamically adapting to input variations, resulting in a representation discrepancy between convolution and self-attention as self-attention calculates attention matrices dynamically. Furthermore, when stacking token mixers that consist of convolution and self-attention to form a deep network, the static nature of convolution hinders the fusion of features previously generated by self-attention into convolution kernels. These two limitations result in a sub-optimal representation capacity of the constructed networks. To find a solution, we propose a lightweight Dual Dynamic Token Mixer (D-Mixer) that aggregates global information and local details in an input-dependent way. D-Mixer works by applying an efficient global attention module and an input-dependent depthwise convolution separately on evenly split feature segments, endowing the network with strong inductive bias and an enlarged effective receptive field. We use D-Mixer as the basic building block to design TransXNet, a novel hybrid CNN-Transformer vision backbone network that delivers compelling performance. In the ImageNet-1K image classification task, TransXNet-T surpasses Swin-T by 0.3% in top-1 accuracy while requiring less than half of the computational cost. Furthermore, TransXNet-S and TransXNet-B exhibit excellent model scalability, achieving top-1 accuracy of 83.8% and 84.6% respectively, with reasonable computational costs. Additionally, our proposed network architecture demonstrates strong generalization capabilities in various dense prediction tasks, outperforming other state-of-the-art networks while having lower computational costs. Code is available at <a class="link-external link-https" href="https://github.com/LMMMEng/TransXNet" rel="external noopener nofollow">this https URL</a>.

CNXA: A Novel Attention Mechanism Aided Convolution Network

DCANet: Learning Connected Attentions for Convolutional Neural Networks

An Attention Module for Convolutional Neural Networks

Attention Mechanism Based ResNeXt Network for Automatic Modulation Classification.

HAM: Hybrid Attention Module in Deep Convolutional Neural Networks for Image Classification

A Multi-scale Contextual Attention Mechanism for Convolutional Neural Networks

A Novel Image Super-Resolution Method Based on Attention Mechanism

EAANet: Efficient Attention Augmented Convolutional Networks

KVT: K-Nn Attention for Boosting Vision Transformers.

A Study of ConvNeXt Architectures for Enhanced Image Captioning

A Novel Attention-Based Network for Fast Salient Object Detection

Self-attentional Convolution for Neural Networks

ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation

X-volution: On the unification of convolution and self-attention

An Overview of the Attention Mechanisms in Computer Vision

Convolutional Neural Network optimization via Channel Reassessment Attention module

Convolution-Based Neural Attention with Applications to Sentiment Classification

Improved deep CNNs based on Nonlinear Hybrid Attention Module for image classification

TransXNet: Learning Both Global and Local Dynamics with a Dual Dynamic Token Mixer for Visual Recognition