Abstract: There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which is concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine-grained feature needs to be extracted from a token and its neighbors. While the lack of data hinders ViTs to attend the spatial relevance. Second, on channel aspect, representation exhibits diversity on different channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition. To this end, we propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. On spatial aspect, we adopt a hybrid structure, in which convolution is integrated into patch embedding and multi-layer perceptron module, forcing the model to capture the token features as well as their neighboring features. On channel aspect, we introduce a dynamic feature aggregation module in MLP and a brand new "head token" design in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other. The fusion of weak channel representation forms a strong enough representation for classification. With this design, we successfully eliminate the performance gap between CNNs and ViTs, and our DHVT achieves a series of state-of-the-art performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% on ImageNet-1K with 24.0M parameters. Code is available at https://github.com/ArieSeirack/DHVT.

Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective

GhostViT: Expediting Vision Transformers Via Cheap Operations

Visualization Comparison of Vision Transformers and Convolutional Neural Networks

How to Train Vision Transformer on Small-scale Datasets?

ViR:the Vision Reservoir

Locality Guidance for Improving Vision Transformers on Tiny Datasets.

Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Training Vision Transformers with only 2040 Images.

DeepViT: Towards Deeper Vision Transformer

A Unified and Biologically-Plausible Relational Graph Representation of Vision Transformers

A Unified and Biologically Plausible Relational Graph Representation of Vision Transformers

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training.

How Does Attention Work in Vision Transformers? A Visual Analytics Attempt

Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Images

Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets

CRViT: Vision transformer advanced by causality and inductive bias for image recognition

Add-Vit: CNN-Transformer Hybrid Architecture for Small Data Paradigm Processing

Scaled ReLU Matters for Training Vision Transformers

Retina Vision Transformer (RetinaViT): Introducing Scaled Patches into Vision Transformers

Do Vision Transformers See Like Convolutional Neural Networks?