Abstract:In this paper, we present DetKDS, the first framework that searches for optimal detection distillation policies. Manual design of detection distillers becomes challenging and time-consuming due to significant disparities in distillation behaviors between detectors with different backbones, paradigms, and label assignments. To tackle these challenges, we leverage search algorithms to discover optimal distillers for homogeneous and heterogeneous student-teacher pairs. Firstly, our search space encompasses global features, foreground-background features, instance features, logits response, and localization response as inputs. Then, we construct omni-directional cascaded transformations and obtain the distiller by selecting the advanced distance function and common weight value options. Finally, we present a divide-and-conquer evolutionary algorithm to handle the explosion of the search space. In this strategy, we first evolve the best distiller formulations of individual knowledge inputs and then optimize the combined weights of these multiple distillation losses. DetKDS automates the distillation process without requiring expert design or additional tuning, effectively reducing the teacher-student gap in various scenarios. Based on the analysis of our search results, we provide valuable guidance that contributes to detection distillation designs. Comprehensive experiments on different detectors demonstrate that DetKDS outperforms state-of-the-art methods in detection and instance segmentation tasks. For instance, DetKDS achieves significant gains than baseline detectors: $+3.7$, $+4.1$, $+4.0$, $+3.7$, and $+3.5$ AP on RetinaNet, Faster-RCNN, FCOS, RepPoints, and GFL, respectively. Code at: https://github.com/lliai/DetKDS.

Self-Supervised Keypoint Detection with Distilled Depth Keypoint Representation

Self-Paced Knowledge Distillation for Real-Time Image Guided Depth Completion

X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation

Distilling Object Detectors with Global Knowledge

PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection

Knowledge Distillation with Deep Supervision

TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation

MSSD: multi-scale self-distillation for object detection

LabelDistill: Label-guided Cross-modal Knowledge Distillation for Camera-based 3D Object Detection

SEKD: Self-Evolving Keypoint Detection and Description

Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection

DetKDS: Knowledge Distillation Search for Object Detectors

Repeatable Adaptive Keypoint Detection Via Self-Supervised Learning

Self-Supervised Keypoint Discovery in Behavioral Videos

ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation

Dynamic Knowledge Distillation with Noise Elimination for RGB-D Salient Object Detection

Structured Knowledge Distillation for Accurate and Efficient Object Detection

Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation

MonoSKD: General Distillation Framework for Monocular 3D Object Detection via Spearman Correlation Coefficient

Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution

Instance-Aware Distillation for Efficient Object Detection in Remote Sensing Images