Abstract:This paper presents a Cosine Similarity-Based Knowledge Distillation (CSKD) for robust, lightweight object detectors. Knowledge Distillation (KD) has been effective in enhancing the performance of compact models in image classification by leveraging deep CNN models. However, the complex and multifaceted nature of object detection, characterized by its modular design and multitasking requirements, poses significant challenges for traditional KD techniques. These challenges are further compounded by the conventional reliance on the Mean Squared Error (MSE) loss function and the limited application of enhanced feature representations to the training phase. Addressing these limitations, the proposed CSKD method combines cosine similarity guidance with MSE loss to facilitate a more effective knowledge transfer from the teacher model to the student model. This is achieved by distilling both intermediate features and prediction outputs, aided by an assistant prediction branch designed to learn directly from the teacher's predictions. This dual-faceted distillation strategy enables the student model to better mimic the teacher model's behavior, leading to improved performance. The proposed method demonstrates versatility and robustness across various object detector architectures without the need for additional feature enhancement layers during training. Notably, employing ResNet-50 as the teacher model and ResNet-18 as the student model, we achieve new benchmarks in KD for object detection across several architectures, including Faster-RCNN, RetinaNet, FCOS, and GFL, with respective mAP scores of 36.6, 35.2, 35.9, and 38.9. These results highlights the effectiveness of CSKD in advancing the state-of-the-art in KD for object detection, offering a compelling solution to the challenges previously faced by traditional KD methods in this domain. The code of the proposed CSKD is available at https://github.com/swkdn16/CSKD .

Category-oriented Localization Distillation for SAR Object Detection and A Unified Benchmark

Spatial Likelihood Voting with Self-Knowledge Distillation for Weakly Supervised Object Detection.

SAR-CDSS: A Semi-Supervised Cross-Domain Object Detection from Optical to SAR Domain

SARDet-100K: Towards Open-Source Benchmark and ToolKit for Large-Scale SAR Object Detection

Consistency- and dependence-guided knowledge distillation for object detection in remote sensing images

Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection

Adaptive Knowledge Distillation for Lightweight Remote Sensing Object Detectors Optimizing

Localization Distillation for Object Detection

Efficient Object Detection in Optical Remote Sensing Imagery via Attention-Based Feature Distillation

Cosine similarity-guided knowledge distillation for robust object detectors

Localization, balance and affinity: a stronger multifaceted collaborative salient object detector in remote sensing images

Two-Way Assistant: A Knowledge Distillation Object Detection Method for Remote Sensing Images

UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye View

RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features

Learning Critical Features for Arbitrary-Oriented Object Detection in Remote-Sensing Optical Images

Distilling Object Detectors with Global Knowledge

Domain-invariant Progressive Knowledge Distillation for UAV-based Object Detection

SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector

CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection