Abstract:In this paper, we propose that small models may not need to absorb the cost of pre-training to reap its benefits. Instead, they can capitalize on the astonishing results achieved by modern, enormous models to a surprising degree. We observe that, when distilled on a task from a pre-trained teacher model, a small model can achieve or surpass the performance it would achieve if it was pre-trained then finetuned on that task. To allow this phenomenon to be easily leveraged, we establish a connection reducing knowledge distillation to modern contrastive learning, opening two doors: (1) vastly different model architecture pairings can work for the distillation, and (2) most contrastive learning algorithms rooted in the theory of Noise Contrastive Estimation can be easily applied and used. We demonstrate this paradigm using pre-trained teacher models from open-source model hubs, Transformer and convolution based model combinations, and a novel distillation algorithm that massages the Alignment/Uniformity perspective of contrastive learning by Wang & Isola (2020) into a distillation objective. We choose this flavor of contrastive learning due to its low computational cost, an overarching theme of this work. We also observe that this phenomenon tends not to occur if the task is data-limited. However, this can be alleviated by leveraging yet another scale-inspired development: large, pre-trained generative models for dataset augmentation. Again, we use an open-source model, and our rudimentary prompts are sufficient to boost the small model`s performance. Thus, we highlight a training method for small models that is up to 94% faster than the standard pre-training paradigm without sacrificing performance. For practitioners discouraged from fully utilizing modern foundation datasets for their small models due to the prohibitive scale, we believe our work keeps that door open.

What problem does this paper attempt to address?

This paper explores whether small models need pre-training to achieve good performance. The study found that through the knowledge distillation (KD) method, small models can directly learn task knowledge from large pre-trained models, achieving or exceeding the performance of pre-training and fine-tuning. The paper proposes a new distillation loss function that combines contrastive learning with KD and utilizes pre-training generative models for data augmentation, thereby improving the performance of small models and reducing reliance on large-scale pre-training datasets. This approach allows small models to train 94% faster than standard pre-training methods while maintaining performance.

On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models

DCCD: Reducing Neural Network Redundancy Via Distillation

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains.

On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models

Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones

HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals

Accelerating Dataset Distillation Via Model Augmentation

Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning

DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings

Contrastive Representation Distillation

Leveraging Zero-Shot Prompting for Efficient Language Model Distillation

Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models

Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation

Teacher-Student Complementary Sample Contrastive Distillation

Length-Adaptive Distillation: Customizing Small Language Model for Dynamic Token Pruning.

Establishing a stronger baseline for lightweight contrastive models

Small Scale Data-Free Knowledge Distillation

Prune Your Model Before Distill It

Pre-training Distillation for Large Language Models: A Design Space Exploration