Abstract:Object detection based on Knowledge Distillation can enhance the capabilities and performance of 5G and 6G networks in various domains, such as autonomous vehicles, smart surveillance, and augmented reality. The integration of object detection with Knowledge Distillation techniques is expected to play a pivotal role in realizing the full potential of these networks. This study presents Shared Knowledge Distillation (Shared-KD) as a solution to overcome optimization challenges caused by disparities in cross-layer features between teacher–student networks. The significant gaps in intermediate-level features between teachers and students present a considerable obstacle to the efficacy of distillation. To tackle this issue, we draw inspiration from collaborative learning in real-world education, where teachers work together to prepare lessons and students engage in peer learning. Building upon this concept, our innovative contributions in model construction are highlighted as follows: (1) A teacher knowledge augmentation module: this module is proposed to combine lower-level teacher features, facilitating the knowledge transfer from the teacher to the student. (2) A student mutual learning module is introduced to enable students to learn from each other, mimicking the peer learning concept in collaborative learning. (3) The Teacher Share Module combines lower-level teacher features: the specific functionality of the teacher knowledge augmentation module is described, which involves combining lower-level teacher features. (4) The multi-step transfer process can be easily optimized due to the minimal gap between the features: the proposed approach breaks down the knowledge transfer process into multiple steps, which can be easily optimized due to the minimal gap between the features involved in each step. Shared-KD uses simple feature losses without additional weights in transformation, resulting in an efficient distillation process that can be easily combined with other methods for further improvement. The effectiveness of our approach is validated through experiments on popular tasks such as object detection and instance segmentation.

A Lightweight Pipeline Edge Detection Model Based on Heterogeneous Knowledge Distillation

Self-Paced Knowledge Distillation for Real-Time Image Guided Depth Completion

Research on Knowledge Distillation Algorithm of Object Detection

Knowledge Distillation Method for Surface Defect Detection.

A Novel Visual Transformer for Long-Distance Pipeline Pattern Recognition in Complex Environment

Robust Aerial Person Detection With Lightweight Distillation Network for Edge Deployment

VDKD: A ViT-Based Student-Teacher Knowledge Distillation for Multi-Texture Class Anomaly Detection

Efficient Convolutional Neural Networks With PWK Compression for Gas Pipelines Leak Detection

Multidimensional Information Fusion and Broad Learning System-Based Condition Recognition for Energy Pipeline Safety

Edge-Computing-Based Knowledge Distillation and Multitask Learning for Partial Discharge Recognition

Channel-wise Knowledge Distillation for Dense Prediction

CLRKDNet: Speeding up Lane Detection with Knowledge Distillation

Efficient Visual Fault Detection for Freight Train Braking System via Heterogeneous Self Distillation in the Wild

A Lightweight Approach for Network Intrusion Detection in Industrial Cyber-Physical Systems Based on Knowledge Distillation and Deep Metric Learning

Adaptive Knowledge Distillation for Lightweight Remote Sensing Object Detectors Optimizing

An Integrated Deep Learning Model for Intelligent Recognition of Long-distance Natural Gas Pipeline Features

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

ETLSH-YOLO: An Edge–Real-Time Transmission Line Safety Hazard Detection Method

Shared Knowledge Distillation Network for Object Detection

Efficient knowledge distillation for hybrid models: A vision transformer‐convolutional neural network to convolutional neural network approach for classifying remote sensing images

Instance, Scale, and Teacher Adaptive Knowledge Distillation for Visual Detection in Autonomous Driving