DDK: Distilling Domain Knowledge for Efficient Large Language Models

Jiaheng Liu,Chenchen Zhang,Jinyang Guo,Yuanxing Zhang,Haoran Que,Ken Deng,Zhiqi Bai,Jie Liu,Ge Zhang,Jiakai Wang,Yanan Wu,Congnan Liu,Wenbo Su,Jiamang Wang,Lin Qu,Bo Zheng

2024-07-23

Abstract:Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a high-performing LLM (i.e., the teacher model). Prevailing techniques in LLM distillation typically use a black-box model API to generate high-quality pretrained and aligned datasets, or utilize white-box distillation by altering the loss function to better transfer knowledge from the teacher LLM. However, these methods ignore the knowledge differences between the student and teacher LLMs across domains. This results in excessive focus on domains with minimal performance gaps and insufficient attention to domains with large gaps, reducing overall performance. In this paper, we introduce a new LLM distillation framework called DDK, which dynamically adjusts the composition of the distillation dataset in a smooth manner according to the domain performance differences between the teacher and student models, making the distillation process more stable and effective. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the high computational and storage demands faced by large language models (LLMs) in practical applications. Specifically, the paper proposes a new framework called DDK to optimize the knowledge distillation process. DDK improves the performance of student models by dynamically adjusting the composition of the distillation dataset to better match the performance differences between teacher and student models in different domains.

The main contributions include:
1. **Study on the Impact of Domain-Specific Data Mixing**: For the first time, the impact of domain-specific data mixing on student model performance is studied, effectively transferring the knowledge of the teacher network to the student model.
2. **Smooth Update Mechanism**: A factor smooth update strategy is proposed to enhance the focus on the target domain during the distillation process, effectively stabilizing the domain knowledge-guided sampling process.
3. **Experimental Validation**: Extensive experiments on multiple benchmark datasets demonstrate that the effectiveness and generalization ability of the DDK method significantly outperform existing continual pre-training baselines and other knowledge distillation methods.

Through these improvements, DDK can significantly enhance the performance of student models, especially in domains where the teacher model performs well and the student model is relatively weak.