Abstract:Knowledge distillation (KD) is a prevalent model compression technique in deep learning, aiming to leverage knowledge from a large teacher model to enhance the training of a smaller student model. It has found success in deploying compact deep models in intelligent applications like intelligent transportation, smart health, and distributed intelligence. Current knowledge distillation methods primarily fall into two categories: offline and online knowledge distillation. Offline methods involve a one-way distillation process, transferring unvaried knowledge from teacher to student, while online methods enable the simultaneous training of multiple peer students. However, existing knowledge distillation methods often face challenges where the student may not fully comprehend the teacher's knowledge due to model capacity gaps, and there might be knowledge incongruence among outputs of multiple students without teacher guidance. To address these issues, we propose a novel reciprocal teacher-student learning inspired by human teaching and examining through forward and feedback knowledge distillation (FFKD). Forward knowledge distillation operates offline, while feedback knowledge distillation follows an online scheme. The rationale is that feedback knowledge distillation enables the pre-trained teacher model to receive feedback from students, allowing the teacher to refine its teaching strategies accordingly. To achieve this, we introduce a new weighting constraint to gauge the extent of students' understanding of the teacher's knowledge, which is then utilized to enhance teaching strategies. Experimental results on five visual recognition datasets demonstrate that the proposed FFKD outperforms current state-of-the-art knowledge distillation methods.

Reciprocal Teacher-Student Learning Via Forward and Feedback Knowledge Distillation

Learning to Teach with Student Feedback

Collaborative Knowledge Distillation Via Multiknowledge Transfer.

Multi-target Knowledge Distillation Via Student Self-reflection

Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation

An Embarrassingly Simple Approach for Knowledge Distillation

STKD: Distilling Knowledge From Synchronous Teaching for Efficient Model Compression

Semi-Online Knowledge Distillation

Stage-by-stage Knowledge Distillation

Residual Error Based Knowledge Distillation

Comparative Knowledge Distillation

Collaborative Knowledge Distillation

Improving Knowledge Distillation With a Customized Teacher

Learning from a Lightweight Teacher for Efficient Knowledge Distillation

Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

Skill-transferring Knowledge Distillation Method

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Knowledge Condensation Distillation

Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation

ResKD: Residual-Guided Knowledge Distillation

Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias