Abstract:With the widespread integration of deep learning techniques in the domain of medical image analysis, there is a prevailing consensus regarding their efficacy in handling high-dimensional and intricate medical image data. However, it is imperative to acknowledge that while complex deep models exhibit a remarkable capacity for processing high-dimensional and intricate data, they often necessitate a substantial allocation of computational resources and time. Furthermore, lightweight models, despite their computational efficiency, tend to underperform when compared to their more intricate counterparts in terms of performance. Hence, the prevailing aspiration is to transfer the cognitive prowess of complex models to their lightweight counterparts. Addressing the aforementioned concern, this study proposes a knowledge distillation approach that encompasses joint feature and soft label transfer. It entails the transference of knowledge from the teacher model’s intermediate features and predictive outcomes to the student model. The student model leverages this knowledge to emulate the behavior of the teacher model, thereby enhancing the precision of its own predictions. Building upon this foundation, we introduce a Res-Transformer teacher model based on the U-Net architecture and a student model known as ResU-Net, which is grounded in residual modules. The Res-Transformer teacher model employs multi-layer residual attention during the downsampling process to capture deep-level features of the image. Subsequently, we have incorporated a Multi-layer Perceptual Attention module (MPA) for each skip connection layer, facilitating the integration of hierarchical upsampled information to restore fine-grained details within the feature maps. The ResU-Net student model enhances network stability through the utilization of residual modules and optimizes skip connections to recover any lost image information during convolutional operations. Lastly, we conducted experimental assessments on multiple disease datasets. The results reveal that the ACC of the Res-Transformer model achieves an impressive 96.9%. Furthermore, through the knowledge distillation method, rich knowledge is effectively transferred to the ResU-Net model, resulting in a remarkable ACC improvement of 7.2%.

An improved transfer learning algorithm using integration method and its application in image segmentation and recognition

Research of improving semantic image segmentation based on a feature fusion model

Improved SVM-AdaBoost Stacking Algorithm with ResNet18

Improved Algorithm for Image Classification Datasets Using Federated Transfer Learning.

Multi-Model Fusion Fine-Grained Image Classification Method Based on Migration Learning

Image Semantic Segmentation Method with Fusion of Transposed Convolution and Deep Residual

A Novel Approach to Optimizing Convolutional Neural Networks for Improved Digital Image Segmentation

Application of Improved DNN Algorithm Based on Feature Fusion in Fine-Grained Image Recognition

The application of improved densenet algorithm in accurate image recognition

EfficientFusion: simple and efficient learning with pixel-level fusion for semantic segmentation

Improved Image Classification With Token Fusion

Semantic Image Segmentation with Improved Position Attention and Feature Fusion

Image Semantic Segmentation Fusion of Edge Detection and AFF Attention Mechanism

A Lightweight YOLOv5-Based Model with Feature Fusion and Dilation Convolution for Image Segmentation

ResNet with Global and Local Image Features, Stacked Pooling Block, for Semantic Segmentation.

A Hybrid Model in Transfer Learning Based Pre-Trained Model and a Scale Factor Updating

An Improved Infrared and Visible Image Fusion Using an Adaptive Contrast Enhancement Method and Deep Learning Network with Transfer Learning

Research on Optimization of Image Recognition Algorithm Based on Deep Learning

The Application of a Hybrid Transfer Algorithm Based on a Convolutional Neural Network Model and an Improved Convolution Restricted Boltzmann Machine Model in Facial Expression Recognition

Pay Attention to Convolution Filters: Towards Fast and Accurate Fine-Grained Transfer Learning

Medical Image Classification: Knowledge Transfer Via Residual U-Net and Vision Transformer-Based Teacher-Student Model with Knowledge Distillation