Abstract:Leading open-source large language models (LLMs) such as Llama-3.1-Instruct-405B are extremely capable at generating text, answering questions, and solving a variety of natural language understanding tasks. However, they incur higher inference cost and latency compared to smaller LLMs. Knowledge distillation provides a way to use outputs from these large, capable teacher models to train smaller student models which can be used for inference at lower cost and latency, while retaining comparable accuracy. We investigate the efficacy of distillation using the Llama-3.1-405B-Instruct teacher and the smaller Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct student models. Contributions of this work include (a) We evaluate the generalizability of distillation with the above Llama-3.1 teacher-student pairs across different tasks and datasets (b) We show that using synthetic data during distillation significantly improves the accuracy of 8B and 70B models, and when used with reasoning chains, even matches or surpasses the zero-shot accuracy of 405B model on some datasets (c) We empirically show that distillation enables 8B and 70B models to internalize 405B's reasoning ability by using only standard fine-tuning (without customizing any loss function). This allows cost and latency-efficient student model inference. (d) We show pitfalls in evaluation of distillation, and present task-specific evaluation, including both human and LLM-grading, and ground-truth based traditional accuracy benchmarks. This methodical study brings out the fundamental importance of synthetic data quality in knowledge distillation, and of combining multiple, task-specific ways of accuracy and quality evaluation in assessing the effectiveness of distillation.

Domain-specific knowledge distillation yields smaller and better models for conversational commerce

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains.

Data Distillation for Controlling Specificity in Dialogue Generation.

Learning to Augment for Data-scarce Domain BERT Knowledge Distillation

LightBERT: A Distilled Chinese BERT Model

Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation

XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

Knowledge Distillation from Internal Representations

Extremely Small BERT Models from Mixed-Vocabulary Training

Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks

Mandarin Text-to-Speech Front-End with Lightweight Distilled Convolution Network

Comparative analysis of strategies of knowledge distillation on BERT for text matching

LAD: Layer-Wise Adaptive Distillation for BERT Model Compression

Multi-Grained Knowledge Distillation for Named Entity Recognition

Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation

Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

DDK: Distilling Domain Knowledge for Efficient Large Language Models

Patient Knowledge Distillation for BERT Model Compression