Abstract:In this paper, we introduce a novel knowledge distillation approach for the semantic segmentation task. Unlike previous methods that rely on power-trained teachers or other modalities to provide additional knowledge, our approach does not require complex teacher models or information from extra sensors. Specifically, for the teacher model training, we propose to noise the label and then incorporate it into input to effectively boost the lightweight teacher performance. To ensure the robustness of the teacher model against the introduced noise, we propose a dual-path consistency training strategy featuring a distance loss between the outputs of two paths. For the student model training, we keep it consistent with the standard distillation for simplicity. Our approach not only boosts the efficacy of knowledge distillation but also increases the flexibility in selecting teacher and student models. To demonstrate the advantages of our Label Assisted Distillation (LAD) method, we conduct extensive experiments on five challenging datasets including Cityscapes, ADE20K, PASCAL-VOC, COCO-Stuff 10K, and COCO-Stuff 164K, five popular models: FCN, PSPNet, DeepLabV3, STDC, and OCRNet, and results show the effectiveness and generalization of our approach. We posit that incorporating labels into the input, as demonstrated in our work, will provide valuable insights into related fields. Code is available at <a class="link-external link-https" href="https://github.com/skyshoumeng/Label_Assisted_Distillation" rel="external noopener nofollow">this https URL</a>.

Enhancing Chinese Word Segmentation Via Pseudo Labels for Practicability

Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision

Neural Chinese Word Segmentation with Lexicon and Unlabeled Data via Posterior Regularization

Neural Networks Incorporating Unlabeled and Partially-labeled Data for Cross-domain Chinese Word Segmentation

Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation

Neural Chinese Word Segmentation with Dictionary Knowledge

Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning

Deep Learning for Chinese Word Segmentation and POS Tagging.

Unsupervised Neural Word Segmentation for Chinese Via Segmental Language Modeling

Learning Pseudo Labels for Semi-and-weakly Supervised Semantic Segmentation

Unsupervised Learning helps Supervised Neural Word Segmentation

BERT Meets Chinese Word Segmentation

RethinkCWS: is Chinese Word Segmentation a Solved Task?

Neural Word Segmentation Learning for Chinese

Neural Chinese Word Segmentation as Sequence to Sequence Translation

Enhanced Pseudo-Label Generation with Self-supervised Training for Weakly-supervised Semantic Segmentation

Long Short-Term Memory Neural Networks for Chinese Word Segmentation.

Improving Chinese Word Segmentation Using Partially Annotated Sentences

Is Word Segmentation Necessary for Deep Learning of Chinese Representations?

Improving Cross-Domain Chinese Word Segmentation with Word Embeddings

Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation