Abstract:Object detection based on Knowledge Distillation can enhance the capabilities and performance of 5G and 6G networks in various domains, such as autonomous vehicles, smart surveillance, and augmented reality. The integration of object detection with Knowledge Distillation techniques is expected to play a pivotal role in realizing the full potential of these networks. This study presents Shared Knowledge Distillation (Shared-KD) as a solution to overcome optimization challenges caused by disparities in cross-layer features between teacher–student networks. The significant gaps in intermediate-level features between teachers and students present a considerable obstacle to the efficacy of distillation. To tackle this issue, we draw inspiration from collaborative learning in real-world education, where teachers work together to prepare lessons and students engage in peer learning. Building upon this concept, our innovative contributions in model construction are highlighted as follows: (1) A teacher knowledge augmentation module: this module is proposed to combine lower-level teacher features, facilitating the knowledge transfer from the teacher to the student. (2) A student mutual learning module is introduced to enable students to learn from each other, mimicking the peer learning concept in collaborative learning. (3) The Teacher Share Module combines lower-level teacher features: the specific functionality of the teacher knowledge augmentation module is described, which involves combining lower-level teacher features. (4) The multi-step transfer process can be easily optimized due to the minimal gap between the features: the proposed approach breaks down the knowledge transfer process into multiple steps, which can be easily optimized due to the minimal gap between the features involved in each step. Shared-KD uses simple feature losses without additional weights in transformation, resulting in an efficient distillation process that can be easily combined with other methods for further improvement. The effectiveness of our approach is validated through experiments on popular tasks such as object detection and instance segmentation.

Improving Building Extraction by Using Knowledge Distillation to Reduce the Impact of Label Noise

Research on Knowledge Distillation Algorithm of Object Detection

Knowledge Distillation Method for Surface Defect Detection.

Improve Knowledge Distillation via Label Revision and Data Selection

Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense Prediction

Use of peer role models to increase Pap smear and mammogram screening in Mexican-American and black women.

Evaluation of muscular load of m. biceps brachii in isometric contraction with the use of microvibration.

Remote Sensing Image Scene Classification with Noisy Label Distillation

ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression

Knowledge Distillation Meets Self-Supervision

Shared Knowledge Distillation Network for Object Detection

SDSNet: Building Extraction in High-Resolution Remote Sensing Images Using a Deep Convolutional Network with Cross-Layer Feature Information Interaction Filtering

GEIKD : Self-knowledge distillation based on gated ensemble networks and influences-based label noise removal

Knowledge Augmentation for Distillation: A General and Effective Approach to Enhance Knowledge Distillation

Supervised domain adaptation for building extraction from off-nadir aerial images

KD-Crowd: a knowledge distillation framework for learning from crowds

Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection

Revisiting Knowledge Distillation Via Label Smoothing Regularization

Extracting Buildings from Remote Sensing Images Using a Multitask Encoder-Decoder Network with Boundary Refinement

SAKD: Sparse attention knowledge distillation