Abstract:Recently, remarkable object detection and facial expression recognition (FER) approaches have been made by researchers. However, all of these models are trained and tested on high-resolution images without considering that low-resolution images are more common in practical application. Therefore, to relieve this issue, in this paper we aim to propose a knowledge distillation approach to transfer the learned high-resolution features from a teacher network to a simpler structured student network trained on low-resolution inputs. In our approach, instead of transferring knowledge from the same layers of the teacher and student network, we chose multi-level knowledge of the teacher network to supervise single-level output of the student network. Furthermore, we do not directly use the knowledge from the teacher network, instead, before knowledge transfer we concatenate different level features of the teacher network and figure our what kind of information is important and what is redundant and set different weight value to them. Then we use these knowledge with different weights to guide the output of single layer student network to extract abundant features from the low-resolution images. To evaluate the effectiveness of our proposed approach, we apply this approach to two models for object detection and facial expression recognition tasks. Through our experiments, we find that, in object detection task, the CornerNet achieves an accuracy of 40.6% on the original MC COCO dataset, while this index drops dramatically to only 34.2% on the resolution degraded images. By comparison, our proposed model trained by our knowledge distillation approach achieves 35.4% and 33.4% on the original and resolution degraded datasets, respectively. At the same time, compared to CornerNet the number of layers of the proposed network has been reduced by about 60%. Furthermore, in the task of facial expression recognition and image classification, the similar experimental results can also be observed.

Knowledge Distillation Method for Surface Defect Detection.

Research on Knowledge Distillation Algorithm of Object Detection

Defect Detection Method Based on Knowledge Distillation

A New Knowledge Distillation Network for Incremental Few-Shot Surface Defect Detection

Pull & Push: Leveraging Differential Knowledge Distillation for Efficient Unsupervised Anomaly Detection and Localization

Multi-scale Attention and Dilation Network for Small Defect Detection

Structured Knowledge Distillation for Accurate and Efficient Object Detection

A Multi-scale Normalizing Flow Network for Surface Defects Detection

VDKD: A ViT-Based Student-Teacher Knowledge Distillation for Multi-Texture Class Anomaly Detection

Road Surface Damage Detection Based on Knowledge Distillation

Lightweight defect detection algorithm of tunnel lining based on knowledge distillation

Multistage feature fusion knowledge distillation

Focal and Global Knowledge Distillation for Detectors

Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors.

Cross-Layer Distillation with Semantic Calibration

Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

Channel Distillation: Channel-Wise Attention for Knowledge Distillation

TL-SDD: A Transfer Learning-Based Method for Surface Defect Detection with Few Samples

A Method for Surface Defect Detection Based on Multiscale Feature Fusion and Pyramid Attention

SAKD: Sparse attention knowledge distillation

SemCKD: Semantic Calibration for Cross-Layer Knowledge Distillation