Abstract:Question Answering (QA) systems have been extensively studied in both academia and the research community due to their wide real-world applications. When building such industrial-scale QA applications, we are facing two prominent challenges, i.e., i) lacking a sufficient amount of training data to learn an accurate model and ii) requiring high inference speed for online model serving. There are generally two ways to mitigate the above-mentioned problems. One is to adopt transfer learning to leverage information from other domains; the other is to distill the “dark knowledge” from a large teacher model to small student models. The former usually employs parameter sharing mechanisms for knowledge transfer, but does not utilize the “dark knowledge” of pre-trained large models. The latter usually does not consider the cross-domain information from other domains. We argue that these two types of methods can be complementary to each other. Hence in this work, we provide a new perspective on the potential of the teacher-student paradigm facilitating cross-domain transfer learning, where the teacher and student tasks belong to heterogeneous domains, with the goal to improve the student model’s performance in the target domain. Our framework considers the “dark knowledge” learned from large teacher models and also leverages the adaptive hints to alleviate the domain differences between teacher and student models. Extensive experiments have been conducted on two text matching tasks for retrieval-based QA systems. Results show the proposed method has better performance than the competing methods including the existing state-of-the-art transfer learning methods. We have also deployed our method in an online production system and observed significant improvements compared to the existing approaches in terms of both accuracy and cross-domain robustness.

Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

A Video Question Answering Model Based on Knowledge Distillation.

Patient Knowledge Distillation for BERT Model Compression

AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression

One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

Cross-domain Knowledge Distillation for Retrieval-based Question Answering Systems.

Attentive Student Meets Multi-Task Teacher: Improved Knowledge Distillation for Pretrained Models.

Knowledge Distillation Application Technology for Chinese NLP

Reinforced Multi-Teacher Selection for Knowledge Distillation

DPAL-BERT: A Faster and Lighter Question Answering Model

Pea-KD: Parameter-efficient and Accurate Knowledge Distillation on BERT

Explanation Guided Knowledge Distillation for Pre-trained Language Model Compression

Dynamic Multi Teacher Knowledge Distillation for Semantic Parsing in Kbqa

Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation

XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

Knowledge Distillation with Source-free Unsupervised Domain Adaptation for BERT Model Compression.

Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models

Uncertainty-Driven Knowledge Distillation for Language Model Compression.

HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers

Semi-Online Knowledge Distillation