Abstract:Visual prompt tuning (VPT) is a promising solution incorporating learnable prompt tokens to customize pre-trained models for downstream tasks. However, VPT and its variants often encounter challenges like prompt initialization, prompt length, and subpar performance in self-supervised pretraining, hindering successful contextual adaptation. This study commences by exploring the correlation evolvement between prompts and patch tokens during proficient training. Inspired by the observation that the prompt tokens tend to share high mutual information with patch tokens, we propose initializing prompts with downstream token prototypes. The strategic initialization, a stand-in for the previous initialization, substantially improves performance in fine-tuning. To refine further, we optimize token construction with a streamlined pipeline that maintains excellent performance with almost no increase in computational expenses compared to VPT. Exhaustive experiments show our proposed approach outperforms existing methods by a remarkable margin. For instance, it surpasses full fine-tuning in 19 out of 24 tasks, using less than 0.4% of learnable parameters on the FGVC and VTAB-1K benchmarks. Notably, our method significantly advances the adaptation for self-supervised pretraining, achieving impressive task performance gains of at least 10% to 30%. Besides, the experimental results demonstrate the proposed SPT is robust to prompt lengths and scales well with model capacity and training data size. We finally provide an insightful exploration into the amount of target data facilitating the adaptation of pre-trained models to downstream tasks. The code is available at

Iterative Soft Prompt-Tuning for Unsupervised Domain Adaptation

STT: Soft Template Tuning for Few-Shot Adaptation

Improving Prompt Tuning with Learned Prompting Layers

Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More

IAPT: Instruction-Aware Prompt Tuning for Large Language Models

IAPT: Instance-Aware Prompt Tuning for Large Language Models

Instance-wise Prompt Tuning for Pretrained Language Models

PPT: Pre-trained Prompt Tuning for Few-shot Learning

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models

Revisiting the Power of Prompt for Visual Tuning

On Transferability of Prompt Tuning for Natural Language Processing

LIPT: Improving Prompt Tuning with Late Inception Reparameterization

Dynamic Prompting: A Unified Framework for Prompt Tuning

Towards Unified Prompt Tuning for Few-shot Text Classification

Hard Sample Aware Prompt-Tuning

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

Domain Adaptation via Prompt Learning

InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding

Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Convert Cross-Domain Classification into Few-Shot Learning: A Unified Prompt-Tuning Framework for Unsupervised Domain Adaptation