Abstract:Knowledge distillation is a common and effective method in model compression, which trains a compact student model to mimic the capability of a large teacher model to get superior generalization. Previous works on knowledge distillation are underperforming for challenging tasks such as object detection, compared to the general application of unsophisticated classification tasks. In this paper, we propose that the failure of knowledge distillation on object detection is mainly caused by the imbalance between features of informative and invalid background. Not all background noise is redundant, and the valuable background noise after screening contains relations between foreground and background. Therefore, we propose a novel regional filtering distillation (RFD) algorithm to solve this problem through two modules: region selection and attention-guided distillation. Region selection first filters massive invalid backgrounds and retains knowledge-dense regions on near object anchor locations. Attention-guided distillation further improves distillation performance on object detection tasks by extracting the relations between foreground and background to migrate key features. Extensive experiments on both one-stage and two-stage detectors have been conducted to prove the effectiveness of RFD. For example, RFD improves 2.8% and 2.6% mAP for ResNet50-RetinaNet and ResNet50-FPN student networks on the MS COCO dataset, respectively. We also evaluate our method with the Faster R-CNN model on Pascal VOC and KITTI benchmark, which obtain 1.52% and 4.36% mAP promotions for the ResNet18-FPN student network, respectively. Furthermore, our method increases 5.70% of mAP for MobileNetv2-SSD compared to the original model. The proposed RFD technique performs highly on detection tasks through regional filtering distillation. In the future, we plan to extend it to more challenging task scenarios, such as segmentation.

VRDistill: Vote Refinement Distillation for Efficient Indoor 3D Object Detection

Research on Knowledge Distillation Algorithm of Object Detection

Distilling Focal Knowledge from Imperfect Expert for 3D Object Detection

PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection

Distilling Temporal Knowledge with Masked Feature Reconstruction for 3D Object Detection

SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection

Distilling Object Detectors with Global Knowledge

Focal Distillation from High-Resolution Data to Low-Resolution Data for 3D Object Detection

Instance-Aware Distillation for Efficient Object Detection in Remote Sensing Images

DFD: Distillng the Feature Disparity Differently for Detectors

UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye View

Representation Disparity-aware Distillation for 3D Object Detection

Empowering Object Detection: Unleashing the Potential of Decoupled and Interactive Distillation

Towards Efficient 3D Object Detection with Knowledge Distillation

3D-to-2D Distillation for Indoor Scene Parsing

Regional filtering distillation for object detection

Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection

Focal and Global Knowledge Distillation for Detectors

Structured Knowledge Distillation for Accurate and Efficient Object Detection

AMD: Adaptive Masked Distillation for Object Detection

'Parallel-Circuitized' distillation for dense object detection