Abstract:Prompt learning has emerged as a valuable technique in enhancing vision-language models (VLMs) such as CLIP for downstream tasks in specific domains. Existing work mainly focuses on designing various learning forms of prompts, neglecting the potential of prompts as effective distillers for learning from larger teacher models. In this paper, we introduce an unsupervised domain prompt distillation framework, which aims to transfer the knowledge of a larger teacher model to a lightweight target model through prompt-driven imitation using unlabeled domain images. Specifically, our framework consists of two distinct stages. In the initial stage, we pre-train a large CLIP teacher model using domain (few-shot) labels. After pre-training, we leverage the unique decoupled-modality characteristics of CLIP by pre-computing and storing the text features as class vectors only once through the teacher text encoder. In the subsequent stage, the stored class vectors are shared across teacher and student image encoders for calculating the predicted logits. Further, we align the logits of both the teacher and student models via KL divergence, encouraging the student image encoder to generate similar probability distributions to the teacher through the learnable prompts. The proposed prompt distillation process eliminates the reliance on labeled data, enabling the algorithm to leverage a vast amount of unlabeled images within the domain. Finally, the well-trained student image encoders and pre-stored text features (class vectors) are utilized for inference. To our best knowledge, we are the first to (1) perform unsupervised domain-specific prompt-driven knowledge distillation for CLIP, and (2) establish a practical pre-storing mechanism of text features as shared class vectors between teacher and student. Extensive experiments on 11 datasets demonstrate the effectiveness of our method.

Boosting Prompt-Based Few-Shot Learners Through Out-of-Domain Knowledge Distillation

Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt

PromptKD: Unsupervised Prompt Distillation for Vision-Language Models

PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning

Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation

BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction

Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

Knowledge Distillation of Black-Box Large Language Models

Progressive Network Grafting for Few-Shot Knowledge Distillation

Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification

PromptDA: Label-guided Data Augmentation for Prompt-based Few-shot Learners

Towards Zero-Shot Knowledge Distillation for Natural Language Processing

HybridPrompt: Domain-Aware Prompting for Cross-Domain Few-Shot Learning

RPLKG: Robust Prompt Learning with Knowledge Graph

SMASH: Improving SMAll Language Models' Few-SHot Ability with Prompt-Based Distillation.

Boosting Knowledge Distillation Via Intra-class Logit Distribution Smoothing

Few-shot Class Incremental Learning Via Prompt Transfer and Knowledge Distillation

Dialogue for Prompting: a Policy-Gradient-Based Discrete Prompt Generation for Few-shot Learning

FreeKD: Knowledge Distillation via Semantic Frequency Prompt

PromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuning

Leveraging Zero-Shot Prompting for Efficient Language Model Distillation