Abstract:Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories from teachers, but also to the regularization of soft targets, which is equally or even more important. Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manually-designed regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65\% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.

Towards Fixing Clever-Hans Predictors with Counterfactual Knowledge Distillation

DCCD: Reducing Neural Network Redundancy Via Distillation

De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts

Improving Knowledge Distillation with Teacher's Explanation

Comparative Knowledge Distillation

Improve Knowledge Distillation via Label Revision and Data Selection

Task-Driven Causal Feature Distillation: Towards Trustworthy Risk Prediction

Collaborative Knowledge Distillation

Dynamic Rectification Knowledge Distillation

Highlight Every Step: Knowledge Distillation via Collaborative Teaching

Reciprocal Teacher-Student Learning Via Forward and Feedback Knowledge Distillation

SemCKD: Semantic Calibration for Cross-Layer Knowledge Distillation

Cooperative Knowledge Distillation: A Learner Agnostic Approach

Debiasing Counterfactuals In the Presence of Spurious Correlations

CAKD: A Correlation-Aware Knowledge Distillation Framework Based on Decoupling Kullback-Leibler Divergence

Model-agnostic counterfactual reasoning for identifying and mitigating answer bias in knowledge tracing

Improved Knowledge Distillation via Adversarial Collaboration

Knowledge Distillation with a Precise Teacher and Prediction with Abstention

Adaptively Integrated Knowledge Distillation and Prediction Uncertainty for Continual Learning

Why does Knowledge Distillation work? Rethink its attention and fidelity mechanism

Revisiting Knowledge Distillation Via Label Smoothing Regularization