Abstract:Minimizing the computation complexity is essential for the popularization of deep networks in practical applications. Nowadays, most researches attempt to accelerate deep networks by designing new network structure or compressing the network parameters. Meanwhile, transfer learning techniques such as knowledge distillation are utilized to keep the performance of deep models. In this paper, we focus on accelerating deep models and relieving the computation burden by using low-resolution (LR) images as inputs while maintaining competitive performance, which is rarely researched in the current literature. Deep networks may encounter serious performance degradation when using LR inputs because many details are unavailable from LR images. Besides, the existing approaches may fail to learn discriminative features for LR images because of the dramatic appearance variations between LR and high-resolution (HR) images. To tackle with the above problems, we propose a resolution-aware knowledge distillation (RKD) framework to narrow the cross-resolution variations by transferring knowledge from HR domain to LR domain. The proposed framework consists of a HR teacher network and a LR student network. First, we introduce a discriminator and propose an adversarial learning strategy to shrink the variations between inputs with changing resolution. Then we design a cross-resolution knowledge distillation (CRKD) loss to train discriminative student network by exploiting the knowledge of the teacher network. The CRKD loss is consisted of a resolution-aware distillation loss, a pair-wise constraint, and a maximum mean discrepancy loss. Experimental results on person re-identification, image classification, face recognition, and defect segmentation tasks demonstrate that RKD outperforms traditional knowledge distillation method by achieving better performance with lower computation complexities. Furthermore, CRKD surpasses the state-of-the-art knowledge distillation methods in transferring knowledge across different resolutions under RKD framework, especially when coping with large resolution differences.

Inter-Region Affinity Distillation for Road Marking Segmentation

Efficient knowledge distillation for liver CT segmentation using growing assistant network

Relative Difficulty Distillation for Semantic Segmentation

Channel Affinity Knowledge Distillation for Semantic Segmentation

Collaborative spatial-temporal distillation for efficient video deraining

Building and road detection from remote sensing images based on weights adaptive multi-teacher collaborative distillation using a fused knowledge

Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data

Knowledge Distillation for Traversable Region Detection of LiDAR Scan in Off-Road Environments

FCKDNet: A Feature Condensation Knowledge Distillation Network for Semantic Segmentation

Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph

TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation

Restructuring the Teacher and Student in Self-Distillation

Cross-Layer Patch Alignment and Intra-and-Inter Patch Relations for Knowledge Distillation

Resolution-Aware Knowledge Distillation for Efficient Inference.

MSKD: Structured knowledge distillation for efficient medical image segmentation

Inter-image Discrepancy Knowledge Distillation for Semantic Segmentation.

LAKD-Activation Mapping Distillation Based on Local Learning

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

ResKD: Residual-Guided Knowledge Distillation

Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation

Adaptive Cross-Architecture Mutual Knowledge Distillation