Abstract:Semantic segmentation benchmarks in the realm of autonomous driving are dominated by large pre-trained transformers, yet their widespread adoption is impeded by substantial computational costs and prolonged training durations. To lift this constraint, we look at efficient semantic segmentation from a perspective of comprehensive knowledge distillation and aim to bridge the gap between multi-source knowledge extractions and transformer-specific patch embeddings. We put forward the Transformer-based Knowledge Distillation (TransKD) framework which learns compact student transformers by distilling both feature maps and patch embeddings of large teacher transformers, bypassing the long pre-training process and reducing the FLOPs by >85.0%. Specifically, we propose two fundamental modules to realize feature map distillation and patch embedding distillation, respectively: (1) Cross Selective Fusion (CSF) enables knowledge transfer between cross-stage features via channel attention and feature map distillation within hierarchical transformers; (2) Patch Embedding Alignment (PEA) performs dimensional transformation within the patchifying process to facilitate the patch embedding distillation. Furthermore, we introduce two optimization modules to enhance the patch embedding distillation from different perspectives: (1) Global-Local Context Mixer (GL-Mixer) extracts both global and local information of a representative embedding; (2) Embedding Assistant (EA) acts as an embedding method to seamlessly bridge teacher and student models with the teacher's number of channels. Experiments on Cityscapes, ACDC, NYUv2, and Pascal VOC2012 datasets show that TransKD outperforms state-of-the-art distillation frameworks and rivals the time-consuming pre-training method. The source code is publicly available at <a class="link-external link-https" href="https://github.com/RuipingL/TransKD" rel="external noopener nofollow">this https URL</a>.

Multi-view knowledge distillation for efficient semantic segmentation

Multi-Knowledge Aggregation and Transfer for Semantic Segmentation

Triple-View Knowledge Distillation for Semi-Supervised Semantic Segmentation

A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation

Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation

Channel-wise Distillation for Semantic Segmentation.

Adaptive Perspective Distillation for Semantic Segmentation

Unpaired Multi-modal Segmentation via Knowledge Distillation

TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation

Semantic segmentation method for continuous images based on multi-level knowledge distillation

Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation

Channel-spatial knowledge distillation for efficient semantic segmentation

Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection

Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation

FCKDNet: A Feature Condensation Knowledge Distillation Network for Semantic Segmentation

Normalized Feature Distillation for Semantic Segmentation

Local structure consistency and pixel-correlation distillation for compact semantic segmentation

Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data

A Lightweight and Effective Multi-View Knowledge Distillation Framework for Text-Image Retrieval

Bridging Knowledge Distillation Gap for Few-sample Unsupervised Semantic Segmentation

MSKD: Structured knowledge distillation for efficient medical image segmentation