Abstract:This work introduces a novel knowledge distillation framework for classification tasks where information on existing subclasses is available and taken into consideration. In classification tasks with a small number of classes or binary detection, the amount of information transferred from the teacher to the student is restricted, thus limiting the utility of knowledge distillation. Performance can be improved by leveraging information of possible subclasses within the classes. To that end, we propose the so-called Subclass Knowledge Distillation (SKD), a process of transferring the knowledge of predicted subclasses from a teacher to a smaller student. Meaningful information that is not in the teacher's class logits but exists in subclass logits (e.g., similarities within classes) will be conveyed to the student through the SKD, which will then boost the student's performance. Analytically, we measure how much extra information the teacher can provide the student via the SKD to demonstrate the efficacy of our work. The framework developed is evaluated in clinical application, namely colorectal polyp binary classification. It is a practical problem with two classes and a number of subclasses per class. In this application, clinician-provided annotations are used to define subclasses based on the annotation label's variability in a curriculum style of learning. A lightweight, low-complexity student trained with the SKD framework achieves an F1-score of 85.05%, an improvement of 1.47%, and a 2.10% gain over the student that is trained with and without conventional knowledge distillation, respectively. The 2.10% F1-score gap between students trained with and without the SKD can be explained by the extra subclass knowledge, i.e., the extra 0.4656 label bits per sample that the teacher can transfer in our experiment.

Reinforced Knowledge Distillation: Multi-class Imbalanced Classifier Based on Policy Gradient Reinforcement Learning.

Enhancing Class-Imbalanced Learning with Pre-Trained Guidance Through Class-Conditional Knowledge Distillation

Reinforcement Learning-Based Cost-Sensitive Classifier for Imbalanced Fault Classification

Boosting Knowledge Distillation Via Intra-class Logit Distribution Smoothing

Deep Reinforcement Learning for Imbalanced Classification

Rebalancing network with knowledge stability for class incremental learning

Reinforced Multi-Teacher Selection for Knowledge Distillation

Knowledge Distillation from Single to Multi Labels: an Empirical Study

Partial Knowledge Distillation for Alleviating the Inherent Inter-Class Discrepancy in Federated Learning

Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling

Subclass Knowledge Distillation with Known Subclass Labels

Multiloss Joint Gradient Control Knowledge Distillation for Image Classification

Learn from Balance: Rectifying Knowledge Transfer for Long-Tailed Scenarios

Online Knowledge Distillation via Multi-branch Diversity Enhancement

A Unified Asymmetric Knowledge Distillation Framework for Image Classification

Knowledge Distillation with a Precise Teacher and Prediction with Abstention

Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification

Knowledge Distillation Under Ideal Joint Classifier Assumption

Class-aware Information for Logit-based Knowledge Distillation

Adversarial Distillation for Learning with Privileged Provisions