Abstract:In this paper, we propose that small models may not need to absorb the cost of pre-training to reap its benefits. Instead, they can capitalize on the astonishing results achieved by modern, enormous models to a surprising degree. We observe that, when distilled on a task from a pre-trained teacher model, a small model can achieve or surpass the performance it would achieve if it was pre-trained then finetuned on that task. To allow this phenomenon to be easily leveraged, we establish a connection reducing knowledge distillation to modern contrastive learning, opening two doors: (1) vastly different model architecture pairings can work for the distillation, and (2) most contrastive learning algorithms rooted in the theory of Noise Contrastive Estimation can be easily applied and used. We demonstrate this paradigm using pre-trained teacher models from open-source model hubs, Transformer and convolution based model combinations, and a novel distillation algorithm that massages the Alignment/Uniformity perspective of contrastive learning by Wang & Isola (2020) into a distillation objective. We choose this flavor of contrastive learning due to its low computational cost, an overarching theme of this work. We also observe that this phenomenon tends not to occur if the task is data-limited. However, this can be alleviated by leveraging yet another scale-inspired development: large, pre-trained generative models for dataset augmentation. Again, we use an open-source model, and our rudimentary prompts are sufficient to boost the small model`s performance. Thus, we highlight a training method for small models that is up to 94% faster than the standard pre-training paradigm without sacrificing performance. For practitioners discouraged from fully utilizing modern foundation datasets for their small models due to the prohibitive scale, we believe our work keeps that door open.

BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data

Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?

Choosy Babies Need One Coach: Inducing Mode-Seeking Behavior in BabyLlama with Reverse KL Divergence

Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Self-Improving Teacher Cultivates Better Student: Distillation Calibration for Multimodal Large Language Models

On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models

LLAVADI: What Matters For Multimodal Large Language Models Distillation

LLM Pruning and Distillation in Practice: The Minitron Approach

Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation

Llama 2: Open Foundation and Fine-Tuned Chat Models

MiniDisc: Minimal Distillation Schedule for Language Model Compression

Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models

BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation

TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise

GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model

Distilling Efficient Language-Specific Models for Cross-Lingual Transfer

Multi-Granularity Semantic Revision for Large Language Model Distillation