Abstract:Semantic segmentation benchmarks in the realm of autonomous driving are dominated by large pre-trained transformers, yet their widespread adoption is impeded by substantial computational costs and prolonged training durations. To lift this constraint, we look at efficient semantic segmentation from a perspective of comprehensive knowledge distillation and aim to bridge the gap between multi-source knowledge extractions and transformer-specific patch embeddings. We put forward the Transformer-based Knowledge Distillation (TransKD) framework which learns compact student transformers by distilling both feature maps and patch embeddings of large teacher transformers, bypassing the long pre-training process and reducing the FLOPs by >85.0%. Specifically, we propose two fundamental modules to realize feature map distillation and patch embedding distillation, respectively: (1) Cross Selective Fusion (CSF) enables knowledge transfer between cross-stage features via channel attention and feature map distillation within hierarchical transformers; (2) Patch Embedding Alignment (PEA) performs dimensional transformation within the patchifying process to facilitate the patch embedding distillation. Furthermore, we introduce two optimization modules to enhance the patch embedding distillation from different perspectives: (1) Global-Local Context Mixer (GL-Mixer) extracts both global and local information of a representative embedding; (2) Embedding Assistant (EA) acts as an embedding method to seamlessly bridge teacher and student models with the teacher's number of channels. Experiments on Cityscapes, ACDC, NYUv2, and Pascal VOC2012 datasets show that TransKD outperforms state-of-the-art distillation frameworks and rivals the time-consuming pre-training method. The source code is publicly available at <a class="link-external link-https" href="https://github.com/RuipingL/TransKD" rel="external noopener nofollow">this https URL</a>.

KDAS: Knowledge Distillation via Attention Supervision Framework for Polyp Segmentation

MSKD: Structured knowledge distillation for efficient medical image segmentation

Efficient knowledge distillation for liver CT segmentation using growing assistant network

Knowledge Distillation for Adaptive MRI Prostate Segmentation Based on Limit-Trained Multi-Teacher Models

DSST: A Dual Student Model Guided Student–teacher Framework for Semi-Supervised Medical Image Segmentation

Prototype Knowledge Distillation for Medical Segmentation with Missing Modality.

FKD-Med: Privacy-Aware, Communication-Optimized Medical Image Segmentation via Federated Learning and Model Lightweighting Through Knowledge Distillation

Prompting Segment Anything Model with Domain-Adaptive Prototype for Generalizable Medical Image Segmentation

DDANet: Dual Decoder Attention Network for Automatic Polyp Segmentation

SAM-EG: Segment Anything Model with Egde Guidance framework for efficient Polyp Segmentation

Rethinking Intermediate Layers design in Knowledge Distillation for Kidney and Liver Tumor Segmentation

Adaptation of Distinct Semantics for Uncertain Areas in Polyp Segmentation

Shape-intensity knowledge distillation for robust medical image segmentation

Multi-Layer Dense Attention Decoder for Polyp Segmentation

TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation

Efficient Biomedical Instance Segmentation via Knowledge Distillation

Adaptive Context Selection for Polyp Segmentation

Polyp-DDPM: Diffusion-Based Semantic Polyp Synthesis for Enhanced Segmentation

SelectiveKD: A semi-supervised framework for cancer detection in DBT through Knowledge Distillation and Pseudo-labeling

Multi-scale Information Sharing and Selection Network with Boundary Attention for Polyp Segmentation

Unpaired Multi-modal Segmentation via Knowledge Distillation