Abstract:Recently, remarkable object detection and facial expression recognition (FER) approaches have been made by researchers. However, all of these models are trained and tested on high-resolution images without considering that low-resolution images are more common in practical application. Therefore, to relieve this issue, in this paper we aim to propose a knowledge distillation approach to transfer the learned high-resolution features from a teacher network to a simpler structured student network trained on low-resolution inputs. In our approach, instead of transferring knowledge from the same layers of the teacher and student network, we chose multi-level knowledge of the teacher network to supervise single-level output of the student network. Furthermore, we do not directly use the knowledge from the teacher network, instead, before knowledge transfer we concatenate different level features of the teacher network and figure our what kind of information is important and what is redundant and set different weight value to them. Then we use these knowledge with different weights to guide the output of single layer student network to extract abundant features from the low-resolution images. To evaluate the effectiveness of our proposed approach, we apply this approach to two models for object detection and facial expression recognition tasks. Through our experiments, we find that, in object detection task, the CornerNet achieves an accuracy of 40.6% on the original MC COCO dataset, while this index drops dramatically to only 34.2% on the resolution degraded images. By comparison, our proposed model trained by our knowledge distillation approach achieves 35.4% and 33.4% on the original and resolution degraded datasets, respectively. At the same time, compared to CornerNet the number of layers of the proposed network has been reduced by about 60%. Furthermore, in the task of facial expression recognition and image classification, the similar experimental results can also be observed.

Task Decoupled Knowledge Distillation for Lightweight Face Detectors.

Research on Knowledge Distillation Algorithm of Object Detection

Task Integration Distillation for Object Detectors

Distilling Object Detectors via Decoupled Features

Focal and Global Knowledge Distillation for Detectors

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

Task-Balanced Distillation for Object Detection

Learning Efficient Detector with Semi-supervised Adaptive Distillation

A Multi-Teacher Assisted Knowledge Distillation Approach for Enhanced Face Image Authentication.

Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

Empowering Object Detection: Unleashing the Potential of Decoupled and Interactive Distillation

G-detkd: Towards general distillation framework for object detectors via contrastive and semantic-guided feature imitation

DFD: Distillng the Feature Disparity Differently for Detectors

Structured Knowledge Distillation for Accurate and Efficient Object Detection

Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors.

Hands-on Guidance for Distilling Object Detectors

Grouped Knowledge Distillation for Deep Face Recognition

Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation

Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection

Training Deep Face Recognition for Efficient Inference by Distillation and Mutual Learning

DMKD: Improving Feature-based Knowledge Distillation for Object Detection Via Dual Masking Augmentation