Abstract:Background Recently, many studies have explored fusing features extracted from Convolutional neural networks (CNNs) and transformers to integrate multi‐scale representations for better performance in medical image segmentation tasks. Although these hybrid models have achieved better results than previous CNN‐based and transformer‐based methods, they suffer from high computation and space complexities. Purpose The purpose of this research is to address the prohibitive computation and space complexities of hybrid models, which limit their application in clinical practice where computational resources are usually constrained. Methods We propose a novel model equipped with a dual distillation scheme to sufficiently harness the complementary advantages of CNNs and transformers without compromising model efficiency. We further propose a multi‐scale prior‐knowledge distillation (MPD) module to effectively distill multi‐scale knowledge from features extracted from transformers. In addition, to cooperate with the knowledge distillation scheme, we also propose an efficient and robust Selective Fusion module in the student network. Results We extensively evaluate the proposed model against fourteen different network frameworks on two representative datasets: SipakMed and ISIC 2017. In the SipakMed dataset, 3037 Pap smear images are used for training and 1012 for testing. In the ISIC 2017 dataset, 2000 dermoscopic images are used for training, 150 for validation, and 600 for testing. Experimental results demonstrate that our method not only surpasses existing methods by a considerable margin with respect to the evaluation metrics of mean Intersection over Union, mean Dice coefficient, mean average symmetric surface distance, but also requires fewer computational resources in terms of model parameters and floating‐point operations per second. Conclusions Comprehensive comparisons in terms of segmentation accuracy and computational complexity unequivocally confirm that our method effectively and efficiently integrates the advantages of both CNNs and transformers, showing its suitability and significance for clinical applications.

L3DMC: Lifelong Learning using Distillation via Mixed-Curvature Space

M2Distill: Multi-Modal Distillation for Lifelong Imitation Learning

PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition

L3DOC: Lifelong 3D Object Classification

Image-to-Lidar Relational Distillation for Autonomous Driving Data

Dealing With Heterogeneous 3D MR Knee Images: A Federated Few-Shot Learning Method With Dual Knowledge Distillation

Cross-Resolution Distillation for Efficient 3D Medical Image Registration.

Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgery

Multi‐scale contextual learning for medical image segmentation via dual distillation

Lifelong-MonoDepth: Lifelong Learning for Multidomain Monocular Metric Depth Estimation

Rethinking Intermediate Layers design in Knowledge Distillation for Kidney and Liver Tumor Segmentation

Distillation Learning Guided by Image Reconstruction for One-Shot Medical Image Segmentation

Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledge

Linear Projections of Teacher Embeddings for Few-Class Distillation

Latent Dataset Distillation with Diffusion Models

Embedded Mutual Learning: A Novel Online Distillation Method Integrating Diverse Knowledge Sources.

Lifelong-MonoDepth: Lifelong Learning for Multi-Domain Monocular Metric Depth Estimation

Online Distillation with Continual Learning for Cyclic Domain Shifts

Hybrid mix-up contrastive knowledge distillation

SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection

Data-Efficient Ranking Distillation for Image Retrieval