Abstract:Knowledge distillation is a common and effective method in model compression, which trains a compact student model to mimic the capability of a large teacher model to get superior generalization. Previous works on knowledge distillation are underperforming for challenging tasks such as object detection, compared to the general application of unsophisticated classification tasks. In this paper, we propose that the failure of knowledge distillation on object detection is mainly caused by the imbalance between features of informative and invalid background. Not all background noise is redundant, and the valuable background noise after screening contains relations between foreground and background. Therefore, we propose a novel regional filtering distillation (RFD) algorithm to solve this problem through two modules: region selection and attention-guided distillation. Region selection first filters massive invalid backgrounds and retains knowledge-dense regions on near object anchor locations. Attention-guided distillation further improves distillation performance on object detection tasks by extracting the relations between foreground and background to migrate key features. Extensive experiments on both one-stage and two-stage detectors have been conducted to prove the effectiveness of RFD. For example, RFD improves 2.8% and 2.6% mAP for ResNet50-RetinaNet and ResNet50-FPN student networks on the MS COCO dataset, respectively. We also evaluate our method with the Faster R-CNN model on Pascal VOC and KITTI benchmark, which obtain 1.52% and 4.36% mAP promotions for the ResNet18-FPN student network, respectively. Furthermore, our method increases 5.70% of mAP for MobileNetv2-SSD compared to the original model. The proposed RFD technique performs highly on detection tasks through regional filtering distillation. In the future, we plan to extend it to more challenging task scenarios, such as segmentation.

Learning Region Features for Object Detection.

Deep Regionlets: Blended Representation and Deep Learning for Generic Object Detection

R-FCN plus plus : Towards Accurate Region-Based Fully Convolutional Networks for Object Detection

Regional filtering distillation for object detection

Salient Object Extraction Based on Region Saliency Ratio

Learning Regions and Descriptors for Fine-grained Recognition

Region Proposal Ranking Via Fusion Feature for Object Detection

Improving Object Detection with Region Similarity Learning

Hierarchical Context Embedding for Region-based Object Detection.

Region-level Active Detector Learning

Feature Selective Networks for Object Detection

Optimization Efficient Open-World Visual Region Recognition

R-FCN: Object Detection via Region-based Fully Convolutional Networks

Category Aggregation Among Region Proposals for Object Detection.

HyperNet: Towards Accurate Region Proposal Generation and Joint Object Detection

3DLG-Detector: 3D Object Detection via Simultaneous Local-Global Feature Learning

Combining Local and Global: Rich and Robust Feature Pooling for Visual Recognition.

Global context aware RCNN for object detection

Pyramid Sub-Region Sensitive Network for Object Detection.

Hierarchical objectness network for region proposal generation and object detection.

Unsupervised Object Discovery and Co-Localization by Deep Descriptor Transforming