Abstract:Knowledge distillation (KD) aims to build a lightweight deep neural network model under the guidance of a large-scale teacher model for model simplicity. Despite improved model efficiency through the KD technique, the performance gap between a teacher model and the trained student model remains significant. This is because the knowledge of the teacher model is not effectively transferred to the student model since the mapping landscape of the large-scale teacher model is not fully explored. To tackle this research gap, we propose a novel H ybrid M ix-up C ontrastive K nowledge D istillation (HMCKD) approach, which facilitates a thorough and reliable mapping solution space exploration in order to significantly improve the performance of the student model. Specifically, we design a hybrid mixing strategy, including image-level mixing and feature-level mixing, to form a smoother mapping landscape as a means to provide a stronger guidance in order to embed its richer dark knowledge from a teacher model to its student model. Additionally, we apply two other strategies, including contrastive learning and top-k guided selection, in order to ensure more effective knowledge transferability from the teacher model. Extensive experiments have proved that our proposed HMCKD approach outperforms state-of-the-art knowledge distillation methods when tested on 6 publicly available datasets, such as CIFAR-100, CIFAR-100-C, STL-10, SVHN, TinyImageNet, and ImageNet. Particularly on CIFAR-100 dataset, the average accuracy of students using HMCKD increased by 1.47%. Further, both the visualization results and similarity quantifications have confirmed the narrowed knowledge gap between the teacher and student models. Our source code is available at https://github.com/lambett/HMCKD .

Bridging the Knowledge Gap via Transformer-Based Multi-Layer Correlation Learning

Knowledge Distillation Via Channel Correlation Structure

Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching

Correlation Guided Multi-teacher Knowledge Distillation.

Adaptive Cross-Architecture Mutual Knowledge Distillation

Knowledge Transfer via Dense Cross-Layer Mutual-Distillation

CAKD: A Correlation-Aware Knowledge Distillation Framework Based on Decoupling Kullback-Leibler Divergence

MKTN: Adversarial-Based Multifarious Knowledge Transfer Network from Complementary Teachers

Collaborative Knowledge Distillation Via Multiknowledge Transfer.

Cross-Architecture Knowledge Distillation

Multi-head Knowledge Distillation for Model Compression

Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

Knowledge Distillation via the Target-aware Transformer

Harmonizing knowledge Transfer in Neural Network with Unified Distillation

Hybrid mix-up contrastive knowledge distillation

TAS: Distilling Arbitrary Teacher and Student via a Hybrid Assistant

A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation

Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model Assistance

Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning

Improved Knowledge Distillation via Full Kernel Matrix Transfer

Deep Collective Knowledge Distillation