Abstract:In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less attention is paid to semantic segmentation. Existing methods heavily rely on data augmentation and memory buffer, which entail high computational resource demands when applying them to handle semantic segmentation that requires to preserve high-resolution feature maps for making dense pixel-wise predictions. In order to address this problem, we present Augmentation-free Dense Contrastive Knowledge Distillation (Af-DCD), a new contrastive distillation learning paradigm to train compact and accurate deep neural networks for semantic segmentation applications. Af-DCD leverages a masked feature mimicking strategy, and formulates a novel contrastive learning loss via taking advantage of tactful feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to a target student model while maintaining training efficiency. Extensive experiments on five mainstream benchmarks with various teacher-student network pairs demonstrate the effectiveness of our approach. For instance, the DeepLabV3-Res18|DeepLabV3-MBV2 model trained by Af-DCD reaches 77.03%|76.38% mIOU on Cityscapes dataset when choosing DeepLabV3-Res101 as the teacher, setting new performance records. Besides that, Af-DCD achieves an absolute mIOU improvement of 3.26%|3.04%|2.75%|2.30%|1.42% compared with individually trained counterpart on Cityscapes|Pascal VOC|Camvid|ADE20K|COCO-Stuff-164K. Code is available at <a class="link-external link-https" href="https://github.com/OSVAI/Af-DCD" rel="external noopener nofollow">this https URL</a>

MTED: Multiple Teachers Ensemble Distillation for Compact Semantic Segmentation

Adaptive Perspective Distillation for Semantic Segmentation

Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation

Multi-Granularity Distillation Scheme Towards Lightweight Semi-Supervised Semantic Segmentation

Local structure consistency and pixel-correlation distillation for compact semantic segmentation

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation

Holistic Weighted Distillation for Semantic Segmentation

Augmentation-free Dense Contrastive Distillation for Efficient Semantic Segmentation

Attention-guided Feature Distillation for Semantic Segmentation

Semi-supervised Semantic Segmentation with Mutual Knowledge Distillation

Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation

Multi-Knowledge Aggregation and Transfer for Semantic Segmentation

Difference-Aware Distillation for Semantic Segmentation

Semantic segmentation method for continuous images based on multi-level knowledge distillation

Self-Decoupling and Ensemble Distillation for Efficient Segmentation.

Channel-wise Distillation for Semantic Segmentation.

Bridging Knowledge Distillation Gap for Few-sample Unsupervised Semantic Segmentation

Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data

Self-Distillation for Robust LiDAR Semantic Segmentation in Autonomous Driving

Relative Difficulty Distillation for Semantic Segmentation