Abstract:In this paper, we propose that small models may not need to absorb the cost of pre-training to reap its benefits. Instead, they can capitalize on the astonishing results achieved by modern, enormous models to a surprising degree. We observe that, when distilled on a task from a pre-trained teacher model, a small model can achieve or surpass the performance it would achieve if it was pre-trained then finetuned on that task. To allow this phenomenon to be easily leveraged, we establish a connection reducing knowledge distillation to modern contrastive learning, opening two doors: (1) vastly different model architecture pairings can work for the distillation, and (2) most contrastive learning algorithms rooted in the theory of Noise Contrastive Estimation can be easily applied and used. We demonstrate this paradigm using pre-trained teacher models from open-source model hubs, Transformer and convolution based model combinations, and a novel distillation algorithm that massages the Alignment/Uniformity perspective of contrastive learning by Wang & Isola (2020) into a distillation objective. We choose this flavor of contrastive learning due to its low computational cost, an overarching theme of this work. We also observe that this phenomenon tends not to occur if the task is data-limited. However, this can be alleviated by leveraging yet another scale-inspired development: large, pre-trained generative models for dataset augmentation. Again, we use an open-source model, and our rudimentary prompts are sufficient to boost the small model`s performance. Thus, we highlight a training method for small models that is up to 94% faster than the standard pre-training paradigm without sacrificing performance. For practitioners discouraged from fully utilizing modern foundation datasets for their small models due to the prohibitive scale, we believe our work keeps that door open.

CoTuning: A Large-Small Model Collaborating Distillation Framework for Better Model Generalization

Co-training and Co-distillation for Quality Improvement and Compression of Language Models

Joint Structured Pruning and Dense Knowledge Distillation for Efficient Transformer Model Compression

Collaborative Distillation for Ultra-Resolution Universal Style Transfer

DCCD: Reducing Neural Network Redundancy Via Distillation

Co-Tuning for Transfer Learning.

A Model Compression Method Using Significant Data and Knowledge Distillation

Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models

Contrastive Distillation on Intermediate Representations for Language Model Compression

On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models

One-stop Training of Multiple Capacity Models

Less is More: Task-aware Layer-wise Distillation for Language Model Compression

CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders

Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA

On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion

HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers

Deep Incubation: Training Large Models by Divide-and-Conquering

Bi-tuning: Efficient Transfer from Pre-trained Models

CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model

Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains

Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation