Abstract:Deep neural models have achieved remarkable performance on various supervised and unsupervised learning tasks, but it is a challenge to deploy these large-size networks on resource-limited devices. As a representative type of model compression and acceleration methods, knowledge distillation (KD) solves this problem by transferring knowledge from heavy teachers to lightweight students. However, most distillation methods focus on imitating the responses of teacher networks but ignore the information redundancy of student networks. In this article, we propose a novel distillation framework difference-based channel contrastive distillation (DCCD), which introduces channel contrastive knowledge and dynamic difference knowledge into student networks for redundancy reduction. At the feature level, we construct an efficient contrastive objective that broadens student networks' feature expression space and preserves richer information in the feature extraction stage. At the final output level, more detailed knowledge is extracted from teacher networks by making a difference between multiview augmented responses of the same instance. We enhance student networks to be more sensitive to minor dynamic changes. With the improvement of two aspects of DCCD, the student network gains contrastive and difference knowledge and reduces its overfitting and redundancy. Finally, we achieve surprising results that the student approaches and even outperforms the teacher in test accuracy on CIFAR-100. We reduce the top-1 error to 28.16% on ImageNet classification and 24.15% for cross-model transfer with ResNet-18. Empirical experiments and ablation studies on popular datasets show that our proposed method can achieve state-of-the-art accuracy compared with other distillation methods.

Class similarity weighted knowledge distillation for few shot incremental learning

Few-shot Class Incremental Learning Via Prompt Transfer and Knowledge Distillation

DCCD: Reducing Neural Network Redundancy Via Distillation

Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning

Uncertainty-Guided Semi-Supervised Few-Shot Class-Incremental Learning With Knowledge Distillation

Progressive Network Grafting for Few-Shot Knowledge Distillation

On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning

Few-Shot Class-Incremental Learning Via Class-Aware Bilateral Distillation

Class Incremental Learning Via Multi-hinge Distillation

Class Incremental Learning with Multi-Teacher Distillation

Self-supervised Knowledge Distillation for Few-shot Learning

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

Enhancing Few-Shot Learning in Lightweight Models via Dual-Faceted Knowledge Distillation

Class incremental learning of remote sensing images based on class similarity distillation

A New Knowledge Distillation for Incremental Object Detection

Few-Shot Class-Incremental Learning Via Relation Knowledge Distillation

Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification

Multi-instance semantic similarity transferring for knowledge distillation

Incremental Scene Classification Using Dual Knowledge Distillation and Classifier Discrepancy on Natural and Remote Sensing Images

Maintaining Discrimination and Fairness in Class Incremental Learning

Contrastive Knowledge-Augmented Self-Distillation Approach for Few-Shot Learning