Abstract:Recently, remarkable object detection and facial expression recognition (FER) approaches have been made by researchers. However, all of these models are trained and tested on high-resolution images without considering that low-resolution images are more common in practical application. Therefore, to relieve this issue, in this paper we aim to propose a knowledge distillation approach to transfer the learned high-resolution features from a teacher network to a simpler structured student network trained on low-resolution inputs. In our approach, instead of transferring knowledge from the same layers of the teacher and student network, we chose multi-level knowledge of the teacher network to supervise single-level output of the student network. Furthermore, we do not directly use the knowledge from the teacher network, instead, before knowledge transfer we concatenate different level features of the teacher network and figure our what kind of information is important and what is redundant and set different weight value to them. Then we use these knowledge with different weights to guide the output of single layer student network to extract abundant features from the low-resolution images. To evaluate the effectiveness of our proposed approach, we apply this approach to two models for object detection and facial expression recognition tasks. Through our experiments, we find that, in object detection task, the CornerNet achieves an accuracy of 40.6% on the original MC COCO dataset, while this index drops dramatically to only 34.2% on the resolution degraded images. By comparison, our proposed model trained by our knowledge distillation approach achieves 35.4% and 33.4% on the original and resolution degraded datasets, respectively. At the same time, compared to CornerNet the number of layers of the proposed network has been reduced by about 60%. Furthermore, in the task of facial expression recognition and image classification, the similar experimental results can also be observed.

A Corporate Public Opinion Classification Method Based on Multi-teacher Knowledge Distillation

Research on Knowledge Distillation Algorithm of Object Detection

Reinforced Multi-Teacher Selection for Knowledge Distillation

Cross-domain knowledge distillation for text classification

Class-aware Information for Logit-based Knowledge Distillation

Knowledge Distillation with a Precise Teacher and Prediction with Abstention

Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching

KD-Crowd: a knowledge distillation framework for learning from crowds

Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains

Efficient Point Cloud Classification via Offline Distillation Framework and Negative-Weight Self-Distillation Technique

Bi-Level Orthogonal Multi-Teacher Distillation

AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inference

One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

Knowledge Distillation from Single to Multi Labels: an Empirical Study

XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

Dynamic Knowledge Distillation for Pre-trained Language Models

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

FedBKD: Heterogenous Federated Learning via Bidirectional Knowledge Distillation for Modulation Classification in IoT-Edge System

Teacher-student collaborative knowledge distillation for image classification

Tree-structured Auxiliary Online Knowledge Distillation

Multi-level knowledge distillation for low-resolution object detection and facial expression recognition