Abstract:Knowledge distillation is a simple yet effective technique for deep model compression, which aims to transfer the knowledge learned by a large teacher model to a small student model. To mimic how the teacher teaches the student, existing knowledge distillation methods mainly adapt an unidirectional knowledge transfer, where the knowledge extracted from different intermedicate layers of the teacher model is used to guide the student model. However, it turns out that the students can learn more effectively through multi-stage learning with a self-reflection in the real-world education scenario, which is nevertheless ignored by current knowledge distillation methods. Inspired by this, we devise a new knowledge distillation framework entitled multi-target knowledge distillation via student self-reflection or MTKD-SSR, which can not only enhance the teacher's ability in unfolding the knowledge to be distilled, but also improve the student's capacity of digesting the knowledge. Specifically, the proposed framework consists of three target knowledge distillation mechanisms: a stage-wise channel distillation (SCD), a stage-wise response distillation (SRD), and a cross-stage review distillation (CRD), where SCD and SRD transfer feature-based knowledge (i.e., channel features) and response-based knowledge (i.e., logits) at different stages, respectively; and CRD encourages the student model to conduct self-reflective learning after each stage by a self-distillation of the response-based knowledge. Experimental results on five popular visual recognition datasets, CIFAR-100, Market-1501, CUB200-2011, ImageNet, and Pascal VOC, demonstrate that the proposed framework significantly outperforms recent state-of-the-art knowledge distillation methods.

Knowledge distillation-driven semi-supervised multi-view classification

Knowledge Distillation from Single to Multi Labels: an Empirical Study

Dual Knowledge Distillation on Multiview Pseudo Labels for Unsupervised Person Re-Identification

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Knowledge Distillation Meets Self-Supervision

Reinforced Knowledge Distillation: Multi-class Imbalanced Classifier Based on Policy Gradient Reinforcement Learning.

Towards Generalized Multi-stage Clustering: Multi-view Self-distillation

Dynamic Knowledge Distillation with Cross-Modality Knowledge Transfer

Cross-Level Multi-Instance Distillation for Self-Supervised Fine-Grained Visual Categorization

Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification

MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation

Research on a Cross-Domain Few-Shot Adaptive Classification Algorithm Based on Knowledge Distillation Technology

MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

The Application of Target Embedding AutoEncoder based on Knowledge Distillation in Multi-View Multi-Label Classification

Multi-target Knowledge Distillation Via Student Self-reflection

Knowledge Distillation Based Semi-supervised Hyperspectral Image Classification

Self-Supervised Discriminative Feature Learning for Deep Multi-View Clustering

Semi-supervised Deep Multi-view Stereo

Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning

Semi-supervised Semantic Segmentation with Mutual Knowledge Distillation

Triple-View Knowledge Distillation for Semi-Supervised Semantic Segmentation