Abstract:Leading open-source large language models (LLMs) such as Llama-3.1-Instruct-405B are extremely capable at generating text, answering questions, and solving a variety of natural language understanding tasks. However, they incur higher inference cost and latency compared to smaller LLMs. Knowledge distillation provides a way to use outputs from these large, capable teacher models to train smaller student models which can be used for inference at lower cost and latency, while retaining comparable accuracy. We investigate the efficacy of distillation using the Llama-3.1-405B-Instruct teacher and the smaller Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct student models. Contributions of this work include (a) We evaluate the generalizability of distillation with the above Llama-3.1 teacher-student pairs across different tasks and datasets (b) We show that using synthetic data during distillation significantly improves the accuracy of 8B and 70B models, and when used with reasoning chains, even matches or surpasses the zero-shot accuracy of 405B model on some datasets (c) We empirically show that distillation enables 8B and 70B models to internalize 405B's reasoning ability by using only standard fine-tuning (without customizing any loss function). This allows cost and latency-efficient student model inference. (d) We show pitfalls in evaluation of distillation, and present task-specific evaluation, including both human and LLM-grading, and ground-truth based traditional accuracy benchmarks. This methodical study brings out the fundamental importance of synthetic data quality in knowledge distillation, and of combining multiple, task-specific ways of accuracy and quality evaluation in assessing the effectiveness of distillation.

Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data

When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?

Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

Choosy Babies Need One Coach: Inducing Mode-Seeking Behavior in BabyLlama with Reverse KL Divergence

Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

MiniLLM: Knowledge Distillation of Large Language Models

Mini Minds: Exploring Bebeshka and Zlata Baby Models

ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation

Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach

Pre-training LLMs using human-like development data corpus

LLAVADI: What Matters For Multimodal Large Language Models Distillation

Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios

Leveraging Zero-Shot Prompting for Efficient Language Model Distillation

Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data

Are BabyLMs Second Language Learners?

Task-agnostic Distillation of Encoder-Decoder Language Models

LLM-Neo: Parameter Efficient Knowledge Distillation for Large Language Models

Sub-goal Distillation: A Method to Improve Small Language Agents