Abstract:Deep neural models have achieved remarkable performance on various supervised and unsupervised learning tasks, but it is a challenge to deploy these large-size networks on resource-limited devices. As a representative type of model compression and acceleration methods, knowledge distillation (KD) solves this problem by transferring knowledge from heavy teachers to lightweight students. However, most distillation methods focus on imitating the responses of teacher networks but ignore the information redundancy of student networks. In this article, we propose a novel distillation framework difference-based channel contrastive distillation (DCCD), which introduces channel contrastive knowledge and dynamic difference knowledge into student networks for redundancy reduction. At the feature level, we construct an efficient contrastive objective that broadens student networks' feature expression space and preserves richer information in the feature extraction stage. At the final output level, more detailed knowledge is extracted from teacher networks by making a difference between multiview augmented responses of the same instance. We enhance student networks to be more sensitive to minor dynamic changes. With the improvement of two aspects of DCCD, the student network gains contrastive and difference knowledge and reduces its overfitting and redundancy. Finally, we achieve surprising results that the student approaches and even outperforms the teacher in test accuracy on CIFAR-100. We reduce the top-1 error to 28.16% on ImageNet classification and 24.15% for cross-model transfer with ResNet-18. Empirical experiments and ablation studies on popular datasets show that our proposed method can achieve state-of-the-art accuracy compared with other distillation methods.

A Similarity-Based Model for Topic Distillation

Topic Discovery Based on LDA_col Model and Topic Significance Re-ranking.

Using Less but Important Information for Feature Distillation

Subsite Retrieval: A Novel Concept for Topic Distillation.

Topic Distillation Via Sub-Site Retrieval

Effective Topic Distillation with Key Resource Pre-Selection

DCCD: Reducing Neural Network Redundancy Via Distillation

Improving Neural Topic Models with Wasserstein Knowledge Distillation

Wisdom of Committee: Distilling from Foundation Model to Specialized Application Model

A Generic Approach for Reproducible Model Distillation

Similarity Voting Based Viewpoint Selection for Volumes

Tree-like Decision Distillation

Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching

PairDistill: Pairwise Relevance Distillation for Dense Retrieval

Online Distillation for Pseudo-Relevance Feedback

HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers

Unbiased, Effective, and Efficient Distillation from Heterogeneous Models for Recommender Systems

Teaching What You Should Teach: A Data-Based Distillation Method

Differentiable Dynamic Channel Association for Knowledge Distillation

Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph

Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation