Abstract:Deep neural models have achieved remarkable performance on various supervised and unsupervised learning tasks, but it is a challenge to deploy these large-size networks on resource-limited devices. As a representative type of model compression and acceleration methods, knowledge distillation (KD) solves this problem by transferring knowledge from heavy teachers to lightweight students. However, most distillation methods focus on imitating the responses of teacher networks but ignore the information redundancy of student networks. In this article, we propose a novel distillation framework difference-based channel contrastive distillation (DCCD), which introduces channel contrastive knowledge and dynamic difference knowledge into student networks for redundancy reduction. At the feature level, we construct an efficient contrastive objective that broadens student networks' feature expression space and preserves richer information in the feature extraction stage. At the final output level, more detailed knowledge is extracted from teacher networks by making a difference between multiview augmented responses of the same instance. We enhance student networks to be more sensitive to minor dynamic changes. With the improvement of two aspects of DCCD, the student network gains contrastive and difference knowledge and reduces its overfitting and redundancy. Finally, we achieve surprising results that the student approaches and even outperforms the teacher in test accuracy on CIFAR-100. We reduce the top-1 error to 28.16% on ImageNet classification and 24.15% for cross-model transfer with ResNet-18. Empirical experiments and ablation studies on popular datasets show that our proposed method can achieve state-of-the-art accuracy compared with other distillation methods.

Localized Knowledge Distillation Helps IoT Devices Provide High-performance Visual Services

DCCD: Reducing Neural Network Redundancy Via Distillation

Research on Knowledge Distillation Algorithm of Object Detection

Enhancing Distributed In-Situ CNN Inference in the Internet of Things

Privacy-Preserving Knowledge Distillation in Latency-Critical Federated Task Offloading for Consumer IoT Networks

Deploy Large-Scale Deep Neural Networks in Resource Constrained IoT Devices with Local Quantization Region

Understanding Sensor Data Using Deep Learning Methods on Resource-Constrained Edge Devices.

DeeperThings: Fully Distributed CNN Inference on Resource-Constrained Edge Devices

Efficient Federated Learning for AIoT Applications Using Knowledge Distillation

CS-CNN: Enabling Robust and Efficient Convolutional Neural Networks Inference for Internet-of-Things Applications

EdgeCI: Distributed Workload Assignment and Model Partitioning for CNN Inference on Edge Clusters

Efficient knowledge distillation for hybrid models: A vision transformer‐convolutional neural network to convolutional neural network approach for classifying remote sensing images

DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters

Knowledge Distillation for U-Net Based Image Denoising

Model Compression for IoT Applications in Industry 4.0 via Multiscale Knowledge Transfer

Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition

Toward Decentralized and Collaborative Deep Learning Inference for Intelligent IoT Devices

Enabling Deep Learning on Edge Devices through Filter Pruning and Knowledge Transfer

Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge

A Knowledge Distillation Method Based on IQE Attention Mechanism for Target Recognition in Sar Imagery.