Abstract: There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which is concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine-grained feature needs to be extracted from a token and its neighbors. While the lack of data hinders ViTs to attend the spatial relevance. Second, on channel aspect, representation exhibits diversity on different channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition. To this end, we propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. On spatial aspect, we adopt a hybrid structure, in which convolution is integrated into patch embedding and multi-layer perceptron module, forcing the model to capture the token features as well as their neighboring features. On channel aspect, we introduce a dynamic feature aggregation module in MLP and a brand new "head token" design in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other. The fusion of weak channel representation forms a strong enough representation for classification. With this design, we successfully eliminate the performance gap between CNNs and ViTs, and our DHVT achieves a series of state-of-the-art performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% on ImageNet-1K with 24.0M parameters. Code is available at https://github.com/ArieSeirack/DHVT.

Training Vision Transformers with only 2040 Images.

How to Train Vision Transformer on Small-scale Datasets?

GhostViT: Expediting Vision Transformers Via Cheap Operations

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Images

Training a Vision Transformer from scratch in less than 24 hours with 1 GPU

Effective Vision Transformer Training: A Data-Centric Perspective

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Super Vision Transformer

HSViT: Horizontally Scalable Vision Transformer

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets

ViR:the Vision Reservoir

Convolutional Embedding Makes Hierarchical Vision Transformer Stronger

Locality Guidance for Improving Vision Transformers on Tiny Datasets.

Not All Images Are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length

GvT: A Graph-based Vision Transformer with Talking-Heads Utilizing Sparsity, Trained from Scratch on Small Datasets

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training.

Optimizing Vision Transformers with Data-Free Knowledge Transfer