Abstract:It is uncertain whether the power of transformer architectures can complement existing convolutional neural networks. A few recent attempts have combined convolution with transformer design through a range of structures in series, where the main contribution of this paper is to explore a parallel design approach. While previous transformed-based approaches need to segment the image into patch-wise tokens, we observe that the multi-head self-attention conducted on convolutional features is mainly sensitive to global correlations and that the performance degrades when these correlations are not exhibited. We propose two parallel modules along with multi-head self-attention to enhance the transformer. For local information, a dynamic local enhancement module leverages convolution to dynamically and explicitly enhance positive local patches and suppress the response to less informative ones. For mid-level structure, a novel unary co-occurrence excitation module utilizes convolution to actively search the local co-occurrence between patches. The parallel-designed Dynamic Unary Convolution in Transformer (DUCT) blocks are aggregated into a deep architecture, which is comprehensively evaluated across essential computer vision tasks in image-based classification, segmentation, retrieval and density estimation. Both qualitative and quantitative results show our parallel convolutional-transformer approach with dynamic and unary convolution outperforms existing series-designed structures.

Deep Convolutional Transform Learning -- Extended version

DeConFuse : A Deep Convolutional Transform based Unsupervised Fusion Framework

Advances in Convolutional Neural Networks

Contrastive Deep Convolutional Transform K-Means Clustering

Deep Descriptor Transforming for Image Co-Localization

Dynamic Unary Convolution in Transformers

Label Consistent Transform Learning for Hyperspectral Image Classification

Deep Augmentation: Self-Supervised Learning with Transformations in Activation Space

Multitask Learning for Time Series Data with 2D Convolution

Learning Deep Classifiers With Deep Features

Deep Convolutional Tables: Deep Learning without Convolutions

Demystify Transformers & Convolutions in Modern Image Deep Networks

Learning Transformation-Invariant Representations for Image Recognition with Drop Transformation Networks.

Deep Feature Consistent Deep Image Transformations: Downscaling, Decolorization and HDR Tone Mapping

Unsupervised Object Discovery and Co-Localization by Deep Descriptor Transforming

DCT-Former: Efficient Self-Attention with Discrete Cosine Transform

Deep Transfer Network: Unsupervised Domain Adaptation

DBCvT: Double Branch Convolutional Transformer for Medical Image Classification

Accelerated MRI With Deep Linear Convolutional Transform Learning

Dilated Convolution with Learnable Spacings: beyond bilinear interpolation

Convolutional Dictionary Pair Learning Network for Image Representation Learning