Abstract:With the widespread integration of deep learning techniques in the domain of medical image analysis, there is a prevailing consensus regarding their efficacy in handling high-dimensional and intricate medical image data. However, it is imperative to acknowledge that while complex deep models exhibit a remarkable capacity for processing high-dimensional and intricate data, they often necessitate a substantial allocation of computational resources and time. Furthermore, lightweight models, despite their computational efficiency, tend to underperform when compared to their more intricate counterparts in terms of performance. Hence, the prevailing aspiration is to transfer the cognitive prowess of complex models to their lightweight counterparts. Addressing the aforementioned concern, this study proposes a knowledge distillation approach that encompasses joint feature and soft label transfer. It entails the transference of knowledge from the teacher model’s intermediate features and predictive outcomes to the student model. The student model leverages this knowledge to emulate the behavior of the teacher model, thereby enhancing the precision of its own predictions. Building upon this foundation, we introduce a Res-Transformer teacher model based on the U-Net architecture and a student model known as ResU-Net, which is grounded in residual modules. The Res-Transformer teacher model employs multi-layer residual attention during the downsampling process to capture deep-level features of the image. Subsequently, we have incorporated a Multi-layer Perceptual Attention module (MPA) for each skip connection layer, facilitating the integration of hierarchical upsampled information to restore fine-grained details within the feature maps. The ResU-Net student model enhances network stability through the utilization of residual modules and optimizes skip connections to recover any lost image information during convolutional operations. Lastly, we conducted experimental assessments on multiple disease datasets. The results reveal that the ACC of the Res-Transformer model achieves an impressive 96.9%. Furthermore, through the knowledge distillation method, rich knowledge is effectively transferred to the ResU-Net model, resulting in a remarkable ACC improvement of 7.2%.

Enhancement and evaluation for deep learning-based classification of volumetric neuroimaging with 3D-to-2D knowledge distillation

Attention-Fused CNN Model Compression with Knowledge Distillation for Brain Tumor Segmentation

Classification and Visualization of Alzheimer’s Disease using Volumetric Convolutional Neural Network and Transfer Learning

2.75D: Boosting learning by representing 3D Medical imaging to 2D features for small data

Moving from 2D to 3D: volumetric medical image classification for rectal cancer staging

Multi-Center Brain Imaging Classification Using a Novel 3D CNN Approach.

Super Images -- A New 2D Perspective on 3D Medical Imaging Analysis

KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI workflow

Efficient Training on Alzheimer's Disease Diagnosis with Learnable Weighted Pooling for 3D PET Brain Image Classification

Performance of a Deep Neural Network Algorithm Based on a Small Medical Image Dataset: Incremental Impact of 3D-to-2D Reformation Combined with Novel Data Augmentation, Photometric Conversion, or Transfer Learning

Leveraging 2D Deep Learning ImageNet-trained models for Native 3D Medical Image Analysis

A Deep Transfer Learning Framework for 3D Brain Imaging Based on Optimal Mass Transport

3D Convolutional Neural Networks for Classification of Functional Connectomes

3D Dense Separated Convolution Module for Volumetric Medical Image Analysis

3D CNN-Residual Neural Network Based Multimodal Medical Image Classification

Anisotropic Hybrid Network For Cross-Dimension Transferable Feature Learning In 3d Medical Images

Medical Image Classification: Knowledge Transfer Via Residual U-Net and Vision Transformer-Based Teacher-Student Model with Knowledge Distillation

An Efficient and Robust 3D Medical Image Classification Approach Based on 3D CNN, Time‐Distributed 2D CNN‐BLSTM Models, and mRMR Feature Selection

A Multichannel 2D Convolutional Neural Network Model for Task-Evoked fMRI Data Classification

3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes.

A 3D Convolutional Neural Network for Volumetric Image Semantic Segmentation