Abstract:With the widespread integration of deep learning techniques in the domain of medical image analysis, there is a prevailing consensus regarding their efficacy in handling high-dimensional and intricate medical image data. However, it is imperative to acknowledge that while complex deep models exhibit a remarkable capacity for processing high-dimensional and intricate data, they often necessitate a substantial allocation of computational resources and time. Furthermore, lightweight models, despite their computational efficiency, tend to underperform when compared to their more intricate counterparts in terms of performance. Hence, the prevailing aspiration is to transfer the cognitive prowess of complex models to their lightweight counterparts. Addressing the aforementioned concern, this study proposes a knowledge distillation approach that encompasses joint feature and soft label transfer. It entails the transference of knowledge from the teacher model’s intermediate features and predictive outcomes to the student model. The student model leverages this knowledge to emulate the behavior of the teacher model, thereby enhancing the precision of its own predictions. Building upon this foundation, we introduce a Res-Transformer teacher model based on the U-Net architecture and a student model known as ResU-Net, which is grounded in residual modules. The Res-Transformer teacher model employs multi-layer residual attention during the downsampling process to capture deep-level features of the image. Subsequently, we have incorporated a Multi-layer Perceptual Attention module (MPA) for each skip connection layer, facilitating the integration of hierarchical upsampled information to restore fine-grained details within the feature maps. The ResU-Net student model enhances network stability through the utilization of residual modules and optimizes skip connections to recover any lost image information during convolutional operations. Lastly, we conducted experimental assessments on multiple disease datasets. The results reveal that the ACC of the Res-Transformer model achieves an impressive 96.9%. Furthermore, through the knowledge distillation method, rich knowledge is effectively transferred to the ResU-Net model, resulting in a remarkable ACC improvement of 7.2%.

Knowledge Distillation of Attention and Residual U-Net: Transfer from Deep to Shallow Models for Medical Image Classification.

Medical Image Classification: Knowledge Transfer Via Residual U-Net and Vision Transformer-Based Teacher-Student Model with Knowledge Distillation

Knowledge Distillation Method for Surface Defect Detection.

Research on Knowledge Distillation Algorithm of Object Detection

DCCD: Reducing Neural Network Redundancy Via Distillation

Attention-Fused CNN Model Compression with Knowledge Distillation for Brain Tumor Segmentation

Efficient knowledge distillation for liver CT segmentation using growing assistant network

ABUS Tumor Segmentation Via Decouple Contrastive Knowledge Distillation.

A Medical Image Segmentation Method Combining Knowledge Distillation and Contrastive Learning

Semantic Segmentation of Medical Images Based on Knowledge Distillation Algorithm

RSKD: Enhanced medical image segmentation via multi-layer, rank-sensitive knowledge distillation in Vision Transformer models

A Novel Deep Learning Model for Medical Image Segmentation with Convolutional Neural Network and Transformer

Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

Strengthening Attention: Knowledge Distillation Via Cross-Layer Feature Fusion for Image Classification

Knowledge Distillation Based on Positive-Unlabeled Classification and Attention Mechanism.

ResKD: Residual-Guided Knowledge Distillation

ClearKD: Clear Knowledge Distillation for Medical Image Classification

Relay Knowledge Distillation for Efficiently Boosting the Performance of Shallow Networks

Efficient Medical Image Segmentation Based on Knowledge Distillation

Evaluating Knowledge Transfer in Neural Network for Medical Images

Class Attention Transfer Based Knowledge Distillation