Abstract:Knowledge distillation is a simple yet effective technique for deep model compression, which aims to transfer the knowledge learned by a large teacher model to a small student model. To mimic how the teacher teaches the student, existing knowledge distillation methods mainly adapt an unidirectional knowledge transfer, where the knowledge extracted from different intermedicate layers of the teacher model is used to guide the student model. However, it turns out that the students can learn more effectively through multi-stage learning with a self-reflection in the real-world education scenario, which is nevertheless ignored by current knowledge distillation methods. Inspired by this, we devise a new knowledge distillation framework entitled multi-target knowledge distillation via student self-reflection or MTKD-SSR, which can not only enhance the teacher's ability in unfolding the knowledge to be distilled, but also improve the student's capacity of digesting the knowledge. Specifically, the proposed framework consists of three target knowledge distillation mechanisms: a stage-wise channel distillation (SCD), a stage-wise response distillation (SRD), and a cross-stage review distillation (CRD), where SCD and SRD transfer feature-based knowledge (i.e., channel features) and response-based knowledge (i.e., logits) at different stages, respectively; and CRD encourages the student model to conduct self-reflective learning after each stage by a self-distillation of the response-based knowledge. Experimental results on five popular visual recognition datasets, CIFAR-100, Market-1501, CUB200-2011, ImageNet, and Pascal VOC, demonstrate that the proposed framework significantly outperforms recent state-of-the-art knowledge distillation methods.

Multi-strategy Knowledge Distillation Based Teacher-Student Framework for Machine Reading Comprehension

An Iterative Multi-Source Mutual Knowledge Transfer Framework for Machine Reading Comprehension

Attention-Guided Answer Distillation for Machine Reading Comprehension.

A Multi-answer Multi-task Framework for Real-world Machine Reading Comprehension.

Multi-Strategy to Improve Reading Comprehension

Residual Connection-Based Multi-Step Reasoning Via Commonsense Knowledge For Multiple Choice Machine Reading Comprehension

Enhancing Machine Comprehension Using Multi-Knowledge Bases and Offline Answer Span Improving System

Multi-choice Dialogue-Based Reading Comprehension with Knowledge and Key Turns.

Multi-Passage Machine Reading Comprehension Through Multi-Task Learning and Dual Verification

Multi-Teacher Knowledge Distillation for Text Image Machine Translation

Multi-target Knowledge Distillation Via Student Self-reflection

Confidence-Aware Multi-Teacher Knowledge Distillation

A Question-centric Multi-experts Contrastive Learning Framework for Improving the Accuracy and Interpretability of Deep Sequential Knowledge Tracing Models

Analysis and Improvement of External Knowledge Usage in Machine Multi-Choice Reading Comprehension Tasks

AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inference

A Multi-teacher Knowledge Distillation Framework for Distantly Supervised Relation Extraction with Flexible Temperature

Reference Knowledgeable Network for Machine Reading Comprehension

Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension

A novel multi-domain machine reading comprehension model with domain interference mitigation

MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models

MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension