Abstract:Prompt Tuning, conditioning on task-specific learned prompt vectors, has emerged as a data-efficient and parameter-efficient method for adapting large pretrained vision-language models to multiple downstream tasks. However, existing approaches usually consider learning prompt vectors for each task independently from scratch, thereby failing to exploit the rich shareable knowledge across different vision-language tasks. In this paper, we propose multitask vision-language prompt tuning (MVLPT), which incorporates cross-task knowledge into prompt tuning for vision-language models. Specifically, (i) we demonstrate the effectiveness of learning a single transferable prompt from multiple source tasks to initialize the prompt for each target task; (ii) we show many target tasks can benefit each other from sharing prompt vectors and thus can be jointly learned via multitask prompt tuning. We benchmark the proposed MVLPT using three representative prompt tuning methods, namely text prompt tuning, visual prompt tuning, and the unified vision-language prompt tuning. Results in 20 vision tasks demonstrate that the proposed approach outperforms all single-task baseline prompt tuning methods, setting the new state-of-the-art on the few-shot ELEVATER benchmarks and cross-task generalization benchmarks. To understand where the cross-task knowledge is most effective, we also conduct a large-scale study on task transferability with 20 vision tasks in 400 combinations for each prompt tuning method. It shows that the most performant MVLPT for each prompt tuning method prefers different task combinations and many tasks can benefit each other, depending on their visual similarity and label similarity. Code is available at <a class="link-external link-https" href="https://github.com/sIncerass/MVLPT" rel="external noopener nofollow">this https URL</a>.

On Transferability of Prompt Tuning for Natural Language Processing

FPT: Improving Prompt Tuning Efficiency Via Progressive Training.

Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator

TransPrompt V2: Transferable Prompt-based Fine-tuning for Few-shot Text Classification

Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More

TransPrompt v2: A Transferable Prompting Framework for Cross-task Text Classification

Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts

Dynamic Prompting: A Unified Framework for Prompt Tuning

TransPrompt - Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification.

Exploring Universal Intrinsic Task Subspace Via Prompt Tuning

Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt

Exploring Low-dimensional Intrinsic Task Subspace Via Prompt Tuning.

LIPT: Improving Prompt Tuning with Late Inception Reparameterization

PanDa: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation

PPT: Pre-trained Prompt Tuning for Few-shot Learning

Approximated Prompt Tuning for Vision-Language Pre-trained Models

Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models

Prompt Tuning for Unified Multimodal Pretrained Models.

Instance-wise Prompt Tuning for Pretrained Language Models

Multitask Vision-Language Prompt Tuning