Abstract:Existing knowledge distillation methods mostly focus on distillation of teacher's prediction and intermediate activation. However, the structured representation, which arguably is one of the most critical ingredients of deep models, is largely overlooked. In this work, we propose a novel {\em \modelname{}} ({\bf\em \shortname{})} method dedicated for distilling representational knowledge semantically from a pretrained teacher to a target student. The key idea is that we leverage the teacher's classifier as a semantic critic for evaluating the representations of both teacher and student and distilling the semantic knowledge with high-order structured information over all feature dimensions. This is accomplished by introducing a notion of cross-network logit computed through passing student's representation into teacher's classifier. Further, considering the set of seen classes as a basis for the semantic space in a combinatorial perspective, we scale \shortname{} to unseen classes for enabling effective exploitation of largely available, arbitrary unlabeled training data. At the problem level, this establishes an interesting connection between knowledge distillation with open-set semi-supervised learning (SSL). Extensive experiments show that our \shortname{} outperforms significantly previous state-of-the-art knowledge distillation methods on both coarse object classification and fine face recognition tasks, as well as less studied yet practically crucial binary network distillation. Under more realistic open-set SSL settings we introduce, we reveal that knowledge distillation is generally more effective than existing Out-Of-Distribution (OOD) sample detection, and our proposed \shortname{} is superior over both previous distillation and SSL competitors. The source code is available at \url{<a class="link-external link-https" href="https://github.com/jingyang2017/SRD" rel="external noopener nofollow">this https URL</a>\_ossl}.

Knowledge Distillation Meets Open-Set Semi-supervised Learning

Knowledge Distillation Meets Open-Set Semi-Supervised Learning

Knowledge Distillation for Road Detection based on cross-model Semi-Supervised Learning

Semi-Online Knowledge Distillation

Self-Supervised Learning with Explorative Knowledge Distillation

Knowledge Distillation with Deep Supervision

Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

Multi-target Knowledge Distillation Via Student Self-reflection

Self-Referenced Deep Learning

Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning

Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks

Small Scale Data-Free Knowledge Distillation

Spherical Knowledge Distillation.

Self-Knowledge Distillation via Progressive Associative Learning

Self-Knowledge Distillation with Learning from Role-Model Samples.

SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector

Knowledge Distillation Meets Self-Supervision

Multi-instance semantic similarity transferring for knowledge distillation

Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation

Revisiting Label Smoothing Regularization with Knowledge Distillation

Teaching Semi-Supervised Classifier Via Generalized Distillation.