Abstract:The Vision Transformer (ViT) leverages the Transformer's encoder to capture global information by dividing images into patches and achieves superior performance across various computer vision tasks. However, the self-attention mechanism of ViT captures the global context from the outset, overlooking the inherent relationships between neighboring pixels in images or videos. Transformers mainly focus on global information while ignoring the fine-grained local details. Consequently, ViT lacks inductive bias during image or video dataset training. In contrast, convolutional neural networks (CNNs), with their reliance on local filters, possess an inherent inductive bias, making them more efficient and quicker to converge than ViT with less data. In this paper, we present a lightweight Depth-Wise Convolution module as a shortcut in ViT models, bypassing entire Transformer blocks to ensure the models capture both local and global information with minimal overhead. Additionally, we introduce two architecture variants, allowing the Depth-Wise Convolution modules to be applied to multiple Transformer blocks for parameter savings, and incorporating independent parallel Depth-Wise Convolution modules with different kernels to enhance the acquisition of local information. The proposed approach significantly boosts the performance of ViT models on image classification, object detection and instance segmentation by a large margin, especially on small datasets, as evaluated on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet for image classification, and COCO for object detection and instance segmentation. The source code can be accessed at <a class="link-external link-https" href="https://github.com/ZTX-100/Efficient_ViT_with_DW" rel="external noopener nofollow">this https URL</a>.

A Simple Convolutional Transfer Neural Networks In Vision Tasks

Transform-Invariant Convolutional Neural Networks for Image Classification and Search

CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extraction

Convolutional Networks with Cross-Layer Neurons for Image Recognition

Convolutional networks and applications in vision

Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition

SimpleCNN-UNet: an Optic Disc Image Segmentation Network Based on Efficient Small-Kernel Convolutions

Compact Convolutional Neural Network Transfer Learning for Small-Scale Image Classification

CSFNet: a compact and efficient convolution-transformer hybrid vision model

Simple convolutional neural network on image classification

A Tensor-based Convolutional Neural Network for Small Dataset Classification

Convolution in Convolution for Network in Network

Contextual Convolutional Networks

CMT: Convolutional Neural Networks Meet Vision Transformers

Pay Attention to Convolution Filters: Towards Fast and Accurate Fine-Grained Transfer Learning

CCNNet: a novel lightweight convolutional neural network and its application in traditional Chinese medicine recognition

On the Large-Scale Transferability of Convolutional Neural Networks.

Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

Convolutional Neural Networks Exploiting Attributes of Biological Neurons

Dual Convolutional Neural Networks for Low-Level Vision

Semi-Supervised Transfer Learning for Convolutional Neural Network Based Chinese Character Recognition