Abstract:The complexity of deep neural network models (DNNs) severely limits their application on devices with limited computing and storage resources. Knowledge distillation (KD) is an attractive model compression technology that can effectively alleviate this problem. Multi-teacher knowledge distillation (MKD) aims to leverage the valuable and diverse knowledge distilled by multiple teacher networks to improve the performance of the student network. Existing approaches typically rely on simple methods such as averaging the prediction logits or using sub-optimal weighting strategies to fuse distilled knowledge from multiple teachers. However, employing these techniques cannot fully reflect the importance of teachers and may even mislead student's learning. To address this issue, we propose a novel Decoupled Multi-Teacher Knowledge Distillation based on Entropy (DE-MKD). DE-MKD decouples the vanilla knowledge distillation loss and assigns adaptive weights to each teacher to reflect its importance based on the entropy of their predictions. Furthermore, we extend the proposed approach to distill the intermediate features from multiple powerful but cumbersome teachers to improve the performance of the lightweight student network. Extensive experiments on the publicly available CIFAR-100 image classification benchmark dataset with various teacher-student network pairs demonstrated the effectiveness and flexibility of our approach. For instance, the VGG8|ShuffleNetV2 model trained by DE-MKD reached 75.25%|78.86% top-one accuracy when choosing VGG13|WRN40-2 as the teacher, setting new performance records. In addition, surprisingly, the distilled student model outperformed the teacher in both teacher-student network pairs.

Knowledge Augmentation for Distillation: A General and Effective Approach to Enhance Knowledge Distillation

Knowledge Distillation Improves Graph Structure Augmentation for Graph Neural Networks

Knowledge Condensation Distillation

Collaborative Knowledge Distillation Via Multiknowledge Transfer.

Role-Wise Data Augmentation for Knowledge Distillation

Comparative Knowledge Distillation

An Embarrassingly Simple Approach for Knowledge Distillation

Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation

Attention and feature transfer based knowledge distillation

Multiple-Stage Knowledge Distillation

Harmonizing knowledge Transfer in Neural Network with Unified Distillation

A Novel Multi-Knowledge Distillation Approach.

DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy

Adaptive Cross-Architecture Mutual Knowledge Distillation

Understanding the Effect of Data Augmentation on Knowledge Distillation

Online Knowledge Distillation via Collaborative Learning

Improving Knowledge Distillation with Teacher's Explanation

Improving Knowledge Distillation With a Customized Teacher

Ability-aware knowledge distillation for resource-constrained embedded devices

Knowledge Diffusion for Distillation

Channel Distillation: Channel-Wise Attention for Knowledge Distillation