Abstract:Background Recently, many studies have explored fusing features extracted from Convolutional neural networks (CNNs) and transformers to integrate multi‐scale representations for better performance in medical image segmentation tasks. Although these hybrid models have achieved better results than previous CNN‐based and transformer‐based methods, they suffer from high computation and space complexities. Purpose The purpose of this research is to address the prohibitive computation and space complexities of hybrid models, which limit their application in clinical practice where computational resources are usually constrained. Methods We propose a novel model equipped with a dual distillation scheme to sufficiently harness the complementary advantages of CNNs and transformers without compromising model efficiency. We further propose a multi‐scale prior‐knowledge distillation (MPD) module to effectively distill multi‐scale knowledge from features extracted from transformers. In addition, to cooperate with the knowledge distillation scheme, we also propose an efficient and robust Selective Fusion module in the student network. Results We extensively evaluate the proposed model against fourteen different network frameworks on two representative datasets: SipakMed and ISIC 2017. In the SipakMed dataset, 3037 Pap smear images are used for training and 1012 for testing. In the ISIC 2017 dataset, 2000 dermoscopic images are used for training, 150 for validation, and 600 for testing. Experimental results demonstrate that our method not only surpasses existing methods by a considerable margin with respect to the evaluation metrics of mean Intersection over Union, mean Dice coefficient, mean average symmetric surface distance, but also requires fewer computational resources in terms of model parameters and floating‐point operations per second. Conclusions Comprehensive comparisons in terms of segmentation accuracy and computational complexity unequivocally confirm that our method effectively and efficiently integrates the advantages of both CNNs and transformers, showing its suitability and significance for clinical applications.

Multi‐scale contextual learning for medical image segmentation via dual distillation

MixFormer: a Mixed CNN-Transformer Backbone for Medical Image Segmentation

RSKD: Enhanced medical image segmentation via multi-layer, rank-sensitive knowledge distillation in Vision Transformer models

Exploring Generalizable Distillation for Efficient Medical Image Segmentation

A Lightweight Multi-Scale Multi-Angle Dynamic Interactive Transformer-CNN Fusion Model for 3D Medical Image Segmentation

MSKD: Structured knowledge distillation for efficient medical image segmentation

Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation

Reliable Mutual Distillation for Medical Image Segmentation under Imperfect Annotations

Unpaired Multi-modal Segmentation via Knowledge Distillation

Sub-pixel multi-scale fusion network for medical image segmentation

Multi-Level Global Context Cross Consistency Model for Semi-Supervised Ultrasound Image Segmentation with Diffusion Model

Efficient Medical Image Segmentation Based on Knowledge Distillation

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Multi-Task Multi-Scale Contrastive Knowledge Distillation for Efficient Medical Image Segmentation

Multi-scale Orthogonal Model CNN-Transformer For Medical Image Segmentation

A Medical Image Segmentation Method Combining Knowledge Distillation and Contrastive Learning

Graph Flow: Cross-layer Graph Flow Distillation for Dual Efficient Medical Image Segmentation

A multibranch and multiscale neural network based on semantic perception for multimodal medical image fusion

MS-TCNet: An effective Transformer–CNN combined network using multi-scale feature learning for 3D medical image segmentation

Lightweight medical image segmentation network with multi-scale feature-guided fusion

Context-aware and local-aware fusion with transformer for medical image segmentation