Abstract:Convolution and self-attention are two powerful techniques for representation learning, and they are usually considered as two peer approaches that are distinct from each other. In this paper, we show that there exists a strong underlying relation between them, in the sense that the bulk of computations of these two paradigms are in fact done with the same operation. Specifically, we first show that a traditional convolution with kernel size k × k can be decomposed into k2 individual 1 × 1 convolutions, followed by shift and summation operations. Then, we interpret the projections of queries, keys, and values in self-attention module as multiple 1 × 1 convolutions, followed by the computation of attention weights and aggregation of the values. Therefore, the first stage of both two modules comprises the similar operation. More importantly, the first stage contributes a dominant computation complexity (square of the channel size) comparing to the second stage. This observation naturally leads to an elegant integration of these two seemingly distinct paradigms, i.e., a mixed model that enjoys the benefit of both self-Attention and Convolution (ACmix), while having minimum compu-tational overhead compared to the pure convolution or self-attention counterpart. Extensive experiments show that our model achieves consistently improved results over com-petitive baselines on image recognition and downstream tasks. Code and pre-trained models will be released at https://github.com/LeapLabTHU/ACmix and https://gitee.com/mindspore/models.

SuperConv: Strengthening the Convolution Kernel Via Weight Sharing

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

On the Integration of Self-Attention and Convolution

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

ACNet: Strengthening the Kernel Skeletons for Powerful CNN Via Asymmetric Convolution Blocks

MixConv: Mixed Depthwise Convolutional Kernels

Multi-scale Convolution Aggregation and Stochastic Feature Reuse for DenseNets

A High Efficient Architecture for Convolution Neural Network Accelerator

A Novel Multiconnected Convolutional Network for Super-Resolution.

Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy

XSepConv: Extremely Separated Convolution

Design of a Generic Dynamically Reconfigurable Convolutional Neural Network Accelerator with Optimal Balance

Balanced Mixture of SuperNets for Learning the CNN Pooling Architecture

XSepConv: Extremely Separated Convolution for Efficient Deep Networks with Large Kernels

AsConvSR: Fast and Lightweight Super-Resolution Network with Assembled Convolutions

Improving Efficiency in Convolutional Neural Network with Multilinear Filters

MF-Conv: A Novel Convolutional Approach Using Bit-Resolution-based Weight Decomposition to Eliminate Multiplications for CNN Acceleration

Deep Anchored Convolutional Neural Networks

Incorporating Transformer Designs into Convolutions for Lightweight Image Super-Resolution

Drastically Reducing the Number of Trainable Parameters in Deep CNNs by Inter-layer Kernel-sharing