Abstract:Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at <a class="link-external link-https" href="https://github.com/OliverRensu/TinyMIM" rel="external noopener nofollow">this https URL</a>.

Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection.

A Transformer-Based Object Detector with Coarse-Fine Crossing Representations

HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling

Revealing the Dark Secrets of Masked Image Modeling

Green Hierarchical Vision Transformer for Masked Image Modeling

Integrally Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection

Improving Pixel-based MIM by Reducing Wasted Modeling Capability

Efficient Decoder-Free Object Detection with Transformers

MIMTracking: Masked image modeling enhanced vision transformer for visual object tracking

An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Training

TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models

Masked Image Residual Learning for Scaling Deeper Vision Transformers

An Extendable, Efficient and Effective Transformer-based Object Detector

Object Detection Via Multi-Scale Token Based on Vision Transformer.

ViDT: An Efficient and Effective Fully Transformer-based Object Detector

MixMIM: Mixed and Masked Image Modeling for Efficient Visual Representation Learning

Exploring Plain Vision Transformer Backbones for Object Detection

Remote Sensing Scene Classification with Masked Image Modeling (MIM)

Improve Supervised Representation Learning with Masked Image Modeling

Benchmarking Detection Transfer Learning with Vision Transformers