Abstract:Visual prompt tuning (VPT) is a promising solution incorporating learnable prompt tokens to customize pre-trained models for downstream tasks. However, VPT and its variants often encounter challenges like prompt initialization, prompt length, and subpar performance in self-supervised pretraining, hindering successful contextual adaptation. This study commences by exploring the correlation evolvement between prompts and patch tokens during proficient training. Inspired by the observation that the prompt tokens tend to share high mutual information with patch tokens, we propose initializing prompts with downstream token prototypes. The strategic initialization, a stand-in for the previous initialization, substantially improves performance in fine-tuning. To refine further, we optimize token construction with a streamlined pipeline that maintains excellent performance with almost no increase in computational expenses compared to VPT. Exhaustive experiments show our proposed approach outperforms existing methods by a remarkable margin. For instance, it surpasses full fine-tuning in 19 out of 24 tasks, using less than 0.4% of learnable parameters on the FGVC and VTAB-1K benchmarks. Notably, our method significantly advances the adaptation for self-supervised pretraining, achieving impressive task performance gains of at least 10% to 30%. Besides, the experimental results demonstrate the proposed SPT is robust to prompt lengths and scales well with model capacity and training data size. We finally provide an insightful exploration into the amount of target data facilitating the adaptation of pre-trained models to downstream tasks. The code is available at

GPT understands, too

Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning

P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

PPT: Pre-trained Prompt Tuning for Few-shot Learning

Dynamic Prompting: A Unified Framework for Prompt Tuning

Context-Tuning: Learning Contextualized Prompts for Natural Language Generation

Masked Siamese Prompt Tuning for Few-Shot Natural Language Understanding

XPrompt: Exploring the Extreme of Prompt Tuning

PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models

FPT: Improving Prompt Tuning Efficiency Via Progressive Training.

SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning

Black-Box Tuning for Language-Model-as-a-Service

LIPT: Improving Prompt Tuning with Late Inception Reparameterization

Towards Unified Prompt Tuning for Few-shot Text Classification

Revisiting the Power of Prompt for Visual Tuning

Exploring Lottery Prompts for Pre-trained Language Models

Reliable Gradient-free and Likelihood-free Prompt Tuning

Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner