Abstract:Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks. However, the potential of such multi-task learners has not been exploited during transfer learning. In this work, we present a universal parameter-efficient transfer learning method, termed Predict-Interpolate Tuning ($\pi$-Tuning), for vision, language, and vision-language tasks. It aggregates the parameters of lightweight task-specific experts learned from similar tasks to aid the target downstream task. The task similarities are predicted in a unified modality-independent space, yielding a scalable graph to demonstrate task relationships. $\pi$-Tuning has several appealing benefits. First, it flexibly explores both intra- and inter-modal transferability between similar tasks to improve the accuracy and robustness of transfer learning, especially in data-scarce scenarios. Second, it offers a systematical solution for transfer learning with multi-task prediction-and-then-interpolation, compatible with diverse types of parameter-efficient experts, such as prompt and adapter. Third, an extensive study of task-level mutual benefits on 14 unimodal and 6 multimodal datasets shows that $\pi$-Tuning surpasses fine-tuning and other parameter-efficient transfer learning methods both in full-shot and low-shot regimes. The task graph also enables an in-depth interpretable analysis of task transferability across modalities. The code will be available at <a class="link-external link-https" href="https://github.com/TencentARC/pi-Tuning" rel="external noopener nofollow">this https URL</a>.

Parameter-efficient Tuning of Large-scale Multimodal Foundation Model

Conditional Prompt Tuning for Multimodal Fusion

Enhanced Transfer Learning with Efficient Modeling and Adaptive Fusion of Knowledge Via Prompt Tuning

Prompt Tuning for Unified Multimodal Pretrained Models.

$π$-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation

Multimodal Infusion Tuning for Large Models

Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

M^2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

CoPL:Parameter-Efficient Collaborative Prompt Learning for Audio-Visual Tasks

MoPE: Parameter-Efficient and Scalable Multimodal Fusion via Mixture of Prompt Experts

Towards a Unified View of Parameter-Efficient Transfer Learning

MPT4LM: Multi-Modal Prompt Tuning Makes Pre-Trained Large Language Models Better Vision-Language Learners

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models

Modular and Parameter-Efficient Multimodal Fusion with Prompting

Efficient Multimodal Fusion Via Interactive Prompting

Prompt Tuning for Generative Multimodal Pretrained Models

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts

One Framework to Rule Them All: Unifying Multimodal Tasks with LLM Neural-Tuning

Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer.