Abstract:Prompt tuning has become a new paradigm for model tuning and it has demonstrated success in natural language pretraining and even vision pretraining. In this work, we explore the transfer of prompt tuning to multimodal pretraining, with a focus on generative multimodal pretrained models, instead of contrastive ones. Specifically, we implement prompt tuning on the unified sequence-to-sequence pretrained model adaptive to both understanding and generation tasks. Experimental results demonstrate that the light-weight prompt tuning can achieve comparable performance with finetuning and surpass other light-weight tuning methods. Besides, in comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks. We further figure out that experimental factors, including the prompt length, prompt depth, and reparameteratization, have great impacts on the model performance, and thus we empirically provide a recommendation for the setups of prompt tuning. Despite the observed advantages, we still find some limitations in prompt tuning, and we correspondingly point out the directions for future studies. Codes are available at \url{<a class="link-external link-https" href="https://github.com/OFA-Sys/OFA" rel="external noopener nofollow">this https URL</a>}

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to explore the effectiveness of prompt tuning in generative multi - modal pre - trained models, as well as its advantages and limitations compared with traditional finetuning. Specifically, the research objectives include: 1. **Verify the effectiveness of prompt tuning**: The research aims to verify whether prompt tuning can be effectively applied to the downstream task transfer of generative multi - modal pre - trained models, especially in comprehension and generation tasks. 2. **Compare prompt tuning with traditional finetuning**: By experimentally comparing the performance of prompt tuning and traditional finetuning, explore the advantages of prompt tuning in terms of computational cost, model robustness, etc. 3. **Analyze factors affecting model performance**: The research deeply explores the influence of experimental factors such as prompt length, prompt depth, and re - parameterization on the final downstream task performance, and provides recommended settings. 4. **Explore future improvement directions**: Although prompt tuning shows certain advantages, the research also points out its limitations and proposes future research directions. ### Main findings - **Effectiveness of prompt tuning**: Experimental results show that for large - scale models, prompt tuning can achieve performance comparable to traditional finetuning, while it performs slightly worse on small - scale models. - **Computational cost and robustness**: The number of adjustable parameters required for prompt tuning is much less than that of traditional finetuning, thus significantly reducing the computational cost. In addition, prompt tuning shows stronger robustness under adversarial attacks. - **Influence of experimental factors**: - **Prompt length**: Longer prompts (such as 64 tokens) usually bring better performance, but overly long prompts may lead to performance degradation. - **Prompt depth**: Embedding prompts into all layers of the encoder and decoder has the best effect, although this will increase the computational cost. - **Re - parameterization**: Using methods such as MLP for re - parameterization brings performance improvement on some datasets, but performs poorly on other datasets. ### Future work Although prompt tuning shows some advantages, there are still some limitations, such as computational cost and training instability. Future research can further optimize the prompt tuning method, explore more efficient prompt designs, and try to apply it to more types of multi - modal pre - trained models. In general, through systematic experiments and analysis, this paper provides important insights into the application of prompt tuning in generative multi - modal pre - trained models and points out the direction of future research.

Prompt Tuning for Generative Multimodal Pretrained Models

Prompt Tuning for Unified Multimodal Pretrained Models.

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models

Unified Vision and Language Prompt Learning

MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models

Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning

Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

Dynamic Prompting: A Unified Framework for Prompt Tuning

Multi-Task Pre-Training of Modular Prompt for Few-Shot Learning

Pro-tuning: Unified Prompt Tuning for Vision Tasks

Conditional Prompt Tuning for Multimodal Fusion

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models

No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence

Multitask Vision-Language Prompt Tuning

Prompt Tuning in Code Intelligence: An Experimental Evaluation

P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks

PPT: Pre-trained Prompt Tuning for Few-shot Learning

Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model

Enhancing Few-Shot Transfer Learning with Optimized Multi-Task Prompt Tuning through Modular Prompt Composition

SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddings