Abstract:Knowledge distillation (KD) is a technique that transfers “dark knowledge” from a deep teacher network (teacher) to a shallow student network (student). Despite significant advances in KD, existing work has not adequately mined two crucial types of knowledge: 1) the knowledge of head categories, which represents the relationship between the target category and its similar categories. Our findings reveal that this highly similar (complex) knowledge is essential for improving student’s performance; and 2) the effectively utilized knowledge of tail categories. Existing studies often treat the non-target categories collectively without sufficiently considering the effectiveness of knowledge from tail categories. To tackle these challenges, we reformulate classical KD (ReKD) into two components: Top- K Inter-class Similar Distillation (TISD) and Non-Top- K Inter-class Discriminability (NTID). Firstly, TISD captures and imparts the knowledge of head categories to the student. Our experimental results have verified that TISD is particularly effective in transferring the knowledge of head categories, even in fine-grained dataset classification. Secondly, we theoretically show that the weighting coefficient of NTID increases with the probability of Top- K , leading to stronger suppression of knowledge transfer for tail categories. This observation explains why difficult samples are more informative than simple ones. To better utilize both types of knowledge, we optimize both TISD and NTID using different weighting coefficients, thereby enhancing the student’s ability to learn this valuable knowledge from both head and tail categories. Furthermore, our extensive experimental results demonstrate that ReKD achieves state-of-the-art performance on various image classification datasets, including CIFAR-100, Tiny-ImageNet, and ImageNet-1K, as well as object detection and instance segmentation using the MS-COCO dataset.

NTCE-KD: Non-Target-Class-Enhanced Knowledge Distillation

DCCD: Reducing Neural Network Redundancy Via Distillation

Improving Knowledge Distillation Via Head and Tail Categories

Rethinking Knowledge Distillation Via Cross-Entropy

Class-aware Information for Logit-based Knowledge Distillation

From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels

Boosting Knowledge Distillation Via Intra-class Logit Distribution Smoothing

Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation

Knowledge Condensation Distillation

Adaptive Explicit Knowledge Transfer for Knowledge Distillation

Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching

DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy

Multi-target Knowledge Distillation Via Student Self-reflection

Improving Knowledge Distillation With a Customized Teacher

Adaptive Multi-Teacher Multi-level Knowledge Distillation

Coordinate Attention Guided Dual-Teacher Adaptive Knowledge Distillation for Image Classification

Channel Distillation: Channel-Wise Attention for Knowledge Distillation

Adaptive Cross-Architecture Mutual Knowledge Distillation

Collaborative Knowledge Distillation

Revisiting Knowledge Distillation Via Label Smoothing Regularization