Abstract:Minimizing the computation complexity is essential for the popularization of deep networks in practical applications. Nowadays, most researches attempt to accelerate deep networks by designing new network structure or compressing the network parameters. Meanwhile, transfer learning techniques such as knowledge distillation are utilized to keep the performance of deep models. In this paper, we focus on accelerating deep models and relieving the computation burden by using low-resolution (LR) images as inputs while maintaining competitive performance, which is rarely researched in the current literature. Deep networks may encounter serious performance degradation when using LR inputs because many details are unavailable from LR images. Besides, the existing approaches may fail to learn discriminative features for LR images because of the dramatic appearance variations between LR and high-resolution (HR) images. To tackle with the above problems, we propose a resolution-aware knowledge distillation (RKD) framework to narrow the cross-resolution variations by transferring knowledge from HR domain to LR domain. The proposed framework consists of a HR teacher network and a LR student network. First, we introduce a discriminator and propose an adversarial learning strategy to shrink the variations between inputs with changing resolution. Then we design a cross-resolution knowledge distillation (CRKD) loss to train discriminative student network by exploiting the knowledge of the teacher network. The CRKD loss is consisted of a resolution-aware distillation loss, a pair-wise constraint, and a maximum mean discrepancy loss. Experimental results on person re-identification, image classification, face recognition, and defect segmentation tasks demonstrate that RKD outperforms traditional knowledge distillation method by achieving better performance with lower computation complexities. Furthermore, CRKD surpasses the state-of-the-art knowledge distillation methods in transferring knowledge across different resolutions under RKD framework, especially when coping with large resolution differences.

Super-resolution Guided Knowledge Distillation for Low-Resolution Image Classification

Research on Knowledge Distillation Algorithm of Object Detection

Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition

Vision Mamba Distillation for Low-resolution Fine-grained Image Classification

Data Upcycling Knowledge Distillation for Image Super-Resolution

Resolution-Aware Knowledge Distillation for Efficient Inference.

Low-Resolution Visual Recognition Via Deep Feature Distillation

Classification-Based Dynamic Network for Efficient Super-Resolution

Fakd: feature-affinity based knowledge distillation for efficient image super-resolution

Low-Resolution Object Recognition with Cross-Resolution Relational Contrastive Distillation

Multi-Scale Information Distillation Network for Efficient Image Super-Resolution

Residual Dense Information Distillation Network for Single Image Super-Resolution

Cross-resolution Face Recognition via Identity-Preserving Network and Knowledge Distillation

Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

Prior Knowledge Distillation Network for Face Super-Resolution

Learning knowledge representation with meta knowledge distillation for single image super-resolution

Hybrid Knowledge Distillation from Intermediate Layers for Efficient Single Image Super-Resolution.

Improving Low-Resolution Image Classification by Super-Resolution with Enhancing High-Frequency Content

Lightweight High-Resolution Remote Sensing Scene Classification via Adaptive Enhanced Knowledge Distillation.

Dynamic feature distillation and pyramid split large kernel attention network for lightweight image super-resolution

DDistill-SR: Reparameterized Dynamic Distillation Network for Lightweight Image Super-Resolution