Abstract:Deep neural models have achieved remarkable performance on various supervised and unsupervised learning tasks, but it is a challenge to deploy these large-size networks on resource-limited devices. As a representative type of model compression and acceleration methods, knowledge distillation (KD) solves this problem by transferring knowledge from heavy teachers to lightweight students. However, most distillation methods focus on imitating the responses of teacher networks but ignore the information redundancy of student networks. In this article, we propose a novel distillation framework difference-based channel contrastive distillation (DCCD), which introduces channel contrastive knowledge and dynamic difference knowledge into student networks for redundancy reduction. At the feature level, we construct an efficient contrastive objective that broadens student networks' feature expression space and preserves richer information in the feature extraction stage. At the final output level, more detailed knowledge is extracted from teacher networks by making a difference between multiview augmented responses of the same instance. We enhance student networks to be more sensitive to minor dynamic changes. With the improvement of two aspects of DCCD, the student network gains contrastive and difference knowledge and reduces its overfitting and redundancy. Finally, we achieve surprising results that the student approaches and even outperforms the teacher in test accuracy on CIFAR-100. We reduce the top-1 error to 28.16% on ImageNet classification and 24.15% for cross-model transfer with ResNet-18. Empirical experiments and ablation studies on popular datasets show that our proposed method can achieve state-of-the-art accuracy compared with other distillation methods.

Reducing Overfitting in Deep Convolutional Neural Networks Using Redundancy Regularizer

DCCD: Reducing Neural Network Redundancy Via Distillation

Reducing Overfitting in Deep Networks by Decorrelating Representations

Accelerating Convolutional Neural Networks by Removing Interspatial and Interkernel Redundancies.

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Wordreg: Mitigating the Gap Between Training and Inference with Worst-Case Drop Regularization

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

SparseConnect: Regularising CNNs on Fully Connected Layers

Reliable Identification of Redundant Kernels for Convolutional Neural Network Compression.

Overfitting Remedy by Sparsifying Regularization on Fully-Connected Layers of CNNs.

KRR-CNN: kernels redundancy reduction in convolutional neural networks

Dropout Reduces Underfitting

Convolutional Neural Networks With Dynamic Regularization

Effective and Efficient Dropout for Deep Convolutional Neural Networks

Kernel Orthogonality does not necessarily imply a Decrease in Feature Map Redundancy in CNNs: Convolutional Similarity Minimization

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

ResNet Structure Simplification with the Convolutional Kernel Redundancy Measure

Improving Model Robustness Against Adversarial Examples with Redundant Fully Connected Layer.

Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint.

LocalDrop: A Hybrid Regularization for Deep Neural Networks

A New Measure of Model Redundancy for Compressed Convolutional Neural Networks