Abstract:With the widespread integration of deep learning techniques in the domain of medical image analysis, there is a prevailing consensus regarding their efficacy in handling high-dimensional and intricate medical image data. However, it is imperative to acknowledge that while complex deep models exhibit a remarkable capacity for processing high-dimensional and intricate data, they often necessitate a substantial allocation of computational resources and time. Furthermore, lightweight models, despite their computational efficiency, tend to underperform when compared to their more intricate counterparts in terms of performance. Hence, the prevailing aspiration is to transfer the cognitive prowess of complex models to their lightweight counterparts. Addressing the aforementioned concern, this study proposes a knowledge distillation approach that encompasses joint feature and soft label transfer. It entails the transference of knowledge from the teacher model’s intermediate features and predictive outcomes to the student model. The student model leverages this knowledge to emulate the behavior of the teacher model, thereby enhancing the precision of its own predictions. Building upon this foundation, we introduce a Res-Transformer teacher model based on the U-Net architecture and a student model known as ResU-Net, which is grounded in residual modules. The Res-Transformer teacher model employs multi-layer residual attention during the downsampling process to capture deep-level features of the image. Subsequently, we have incorporated a Multi-layer Perceptual Attention module (MPA) for each skip connection layer, facilitating the integration of hierarchical upsampled information to restore fine-grained details within the feature maps. The ResU-Net student model enhances network stability through the utilization of residual modules and optimizes skip connections to recover any lost image information during convolutional operations. Lastly, we conducted experimental assessments on multiple disease datasets. The results reveal that the ACC of the Res-Transformer model achieves an impressive 96.9%. Furthermore, through the knowledge distillation method, rich knowledge is effectively transferred to the ResU-Net model, resulting in a remarkable ACC improvement of 7.2%.

Attention Correction Feature and Boundary Constraint Knowledge Distillation for Efficient 3D Medical Image Segmentation

Knowledge Distillation Method for Surface Defect Detection.

Efficient knowledge distillation for liver CT segmentation using growing assistant network

A Medical Image Segmentation Method Combining Knowledge Distillation and Contrastive Learning

Towards Efficient Medical Image Segmentation Via Boundary-Guided Knowledge Distillation.

Attention-Fused CNN Model Compression with Knowledge Distillation for Brain Tumor Segmentation

MSKD: Structured knowledge distillation for efficient medical image segmentation

Efficient skin lesion segmentation with boundary distillation

Patch-Level Knowledge Distillation and Regularization for Missing Modality Medical Image Segmentation.

Efficient Medical Image Segmentation Based on Knowledge Distillation

Efficient Biomedical Instance Segmentation via Knowledge Distillation

Target area distillation and section attention segmentation network for accurate 3D medical image segmentation

ABUS Tumor Segmentation Via Decouple Contrastive Knowledge Distillation.

Exploring Generalizable Distillation for Efficient Medical Image Segmentation

Semantic Segmentation of Medical Images Based on Knowledge Distillation Algorithm

FCKDNet: A Feature Condensation Knowledge Distillation Network for Semantic Segmentation

Medical Image Classification: Knowledge Transfer Via Residual U-Net and Vision Transformer-Based Teacher-Student Model with Knowledge Distillation

ACT-Net: Asymmetric Co-Teacher Network for Semi-supervised Memory-efficient Medical Image Segmentation

Graph Relation Distillation for Efficient Biomedical Instance Segmentation

RSKD: Enhanced medical image segmentation via multi-layer, rank-sensitive knowledge distillation in Vision Transformer models

Dual-feature Fusion Attention Network for Small Object Segmentation.