Abstract:Minimizing the computation complexity is essential for the popularization of deep networks in practical applications. Nowadays, most researches attempt to accelerate deep networks by designing new network structure or compressing the network parameters. Meanwhile, transfer learning techniques such as knowledge distillation are utilized to keep the performance of deep models. In this paper, we focus on accelerating deep models and relieving the computation burden by using low-resolution (LR) images as inputs while maintaining competitive performance, which is rarely researched in the current literature. Deep networks may encounter serious performance degradation when using LR inputs because many details are unavailable from LR images. Besides, the existing approaches may fail to learn discriminative features for LR images because of the dramatic appearance variations between LR and high-resolution (HR) images. To tackle with the above problems, we propose a resolution-aware knowledge distillation (RKD) framework to narrow the cross-resolution variations by transferring knowledge from HR domain to LR domain. The proposed framework consists of a HR teacher network and a LR student network. First, we introduce a discriminator and propose an adversarial learning strategy to shrink the variations between inputs with changing resolution. Then we design a cross-resolution knowledge distillation (CRKD) loss to train discriminative student network by exploiting the knowledge of the teacher network. The CRKD loss is consisted of a resolution-aware distillation loss, a pair-wise constraint, and a maximum mean discrepancy loss. Experimental results on person re-identification, image classification, face recognition, and defect segmentation tasks demonstrate that RKD outperforms traditional knowledge distillation method by achieving better performance with lower computation complexities. Furthermore, CRKD surpasses the state-of-the-art knowledge distillation methods in transferring knowledge across different resolutions under RKD framework, especially when coping with large resolution differences.

Joint architecture and knowledge distillation in CNN for Chinese text recognition

DCCD: Reducing Neural Network Redundancy Via Distillation

Cross Architecture Distillation for Face Recognition

Deep Convolutional Neural Networks Based on Knowledge Distillation for Offline Handwritten Chinese Character Recognition

Cross-Architecture Knowledge Distillation

Pruning-and-distillation: One-stage Joint Compression Framework for CNNs Via Clustering

Refining Architectures of Deep Convolutional Neural Networks

Knowledge Distillation for Machine Translation

Self-Distillation: Towards Efficient and Compact Neural Networks

Design of a Very Compact CNN Classifier for Online Handwritten Chinese Character Recognition Using DropWeight and Global Pooling

A Gradually Distilled CNN for SAR Target Recognition

DTCNet: Transformer-CNN Distillation for Super-Resolution of Remote Sensing Image

DCD: A New Framework for Distillation Learning With Dynamic Curriculum.

Class Attention Transfer Based Knowledge Distillation

Promoting CNNs with Cross-Architecture Knowledge Distillation for Efficient Monocular Depth Estimation

Building Efficient CNN Architecture for Offline Handwritten Chinese Character Recognition

Strengthening Attention: Knowledge Distillation Via Cross-Layer Feature Fusion for Image Classification

Highlight Every Step: Knowledge Distillation via Collaborative Teaching

A Study of Designing Compact Classifiers Using Deep Neural Networks for Online Handwritten Chinese Character Recognition

Resolution-Aware Knowledge Distillation for Efficient Inference.