AlignDet: Aligning Pre-training and Fine-tuning in Object Detection

Ming Li,Jie Wu,Xionghui Wang,Chen Chen,Jie Qin,Xuefeng Xiao,Rui Wang,Min Zheng,Xin Pan

2023-08-13

Abstract:The paradigm of large-scale pre-training followed by downstream fine-tuning has been widely employed in various object detection algorithms. In this paper, we reveal discrepancies in data, model, and task between the pre-training and fine-tuning procedure in existing practices, which implicitly limit the detector's performance, generalization ability, and convergence speed. To this end, we propose AlignDet, a unified pre-training framework that can be adapted to various existing detectors to alleviate the discrepancies. AlignDet decouples the pre-training process into two stages, i.e., image-domain and box-domain pre-training. The image-domain pre-training optimizes the detection backbone to capture holistic visual abstraction, and box-domain pre-training learns instance-level semantics and task-aware concepts to initialize the parts out of the backbone. By incorporating the self-supervised pre-trained backbones, we can pre-train all modules for various detectors in an unsupervised paradigm. As depicted in Figure 1, extensive experiments demonstrate that AlignDet can achieve significant improvements across diverse protocols, such as detection algorithm, model backbone, data setting, and training schedule. For example, AlignDet improves FCOS by 5.3 mAP, RetinaNet by 2.1 mAP, Faster R-CNN by 3.3 mAP, and DETR by 2.3 mAP under fewer epochs.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the inconsistencies in data, models, and tasks between the pre-training and fine-tuning processes in existing object detection algorithms. These issues limit the performance, generalization ability, and convergence speed of detectors. Specifically:

1. **Data Inconsistency**: Most pre-training methods are conducted on datasets centered around single objects, such as ImageNet, while detection datasets (e.g., COCO) usually contain multiple objects of different scales and positions. This difference in data characteristics and domains may cause pre-training to deviate from downstream tasks.

2. **Model Inconsistency**: Current pre-training algorithms mainly focus on parts of the model (e.g., the backbone network). Due to the diversity and complexity of detectors, some key detection components (e.g., RPN and regression heads) remain randomly initialized.

3. **Task Inconsistency**: Existing pre-training methods only consider classification tasks as pre-training tasks, failing to capture object-aware positional context, including tasks such as proposal generation, target assignment, and box regression.

To address these issues, the paper proposes AlignDet, a unified pre-training framework that can adapt to various existing detectors and alleviate these inconsistencies through self-supervised pre-training. AlignDet divides the pre-training process into two stages: image-domain pre-training and box-domain pre-training. Through these two stages, AlignDet achieves significant performance improvements across different detection algorithms, model backbones, data settings, and training schedules. For example, with fewer training epochs, AlignDet can improve the performance of FCOS by 5.3 mAP, RetinaNet by 2.1 mAP, Faster R-CNN by 3.3 mAP, and DETR by 2.3 mAP.