Abstract:Abstract Modeling in computer vision has long been dominated by convolutional neural networks (CNNs). Recently, in light of the excellent performance of self-attention mechanism in the language field, transformers tailored for visual data have drawn significant attention and triumphed over CNNs in various vision tasks. These vision transformers heavily rely on large-scale pre-training to achieve competitive accuracy, which not only hinders the freedom of architectural design in downstream tasks like object detection, but also causes learning bias and domain mismatch in the fine-tuning stages. To this end, we aim to get rid of the “pre-train and fine-tune” paradigm of vision transformer and train transformer based object detector from scratch. Some earlier works in the CNNs era have successfully trained CNNs based detectors without pre-training, unfortunately, their findings do not generalize well when the backbone is switched from CNNs to a vision transformer. Instead of proposing a specific vision transformer based detector, in this work, our goal is to reveal the insights of training vision transformer based detectors from scratch. In particular, we expect those insights to help other researchers and practitioners, and inspire more interesting research in other fields, such as remote sensing, visual-linguistic pre-training, etc. One of the key findings is that both architectural changes and more epochs play critical roles in training vision transformer based detectors from scratch. Experiments on the MS COCO dataset demonstrate that vision transformer based detectors trained from scratch can also achieve similar performance to their counterparts with ImageNet pre-training.

Inverting Visual Representations with Detection Transformers

Training Object Detectors from Scratch: An Empirical Study in the Era of Vision Transformer

CTFCD: Channel Transformer Based on Full Convolutional Decoder for Single Image Deraining

A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Analyzing Vision Transformers for Image Classification in Class Embedding Space

Inverting Visual Representations with Convolutional Networks

Vision Transformers Are Active Learners for Image Copy Detection

IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes

Infrared Small-Dim Target Detection with Transformer under Complex Backgrounds

Intriguing Equivalence Structures of the Embedding Space of Vision Transformers

Efficient Decoder-Free Object Detection with Transformers

Do Vision Transformers See Like Convolutional Neural Networks?

Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects

FP-DETR: Detection Transformer Advanced by Fully Pre-training

Understanding Video Transformers via Universal Concept Discovery

PnP-DETR: Towards Efficient Visual Analysis with Transformers

A Survey of Visual Transformers

Training Strategies for Vision Transformers for Object Detection

Three things everyone should know about Vision Transformers

Image Reconstruction using Enhanced Vision Transformer