Abstract:Parameter-efficient transfer learning (PETL) has attracted significant attention lately, due to the increasing size of pre-trained models and the need to fine-tune (FT) them for superior downstream performance. This community-wide enthusiasm has sparked a plethora of approaches. Nevertheless, a systematic study to understand their performance and suitable application scenarios is lacking, leaving questions like when to apply PETL and which approach to use largely unanswered. In this paper, we conduct a unifying empirical study of representative PETL methods in the context of Vision Transformers. We systematically tune their hyper-parameters to fairly compare their accuracy on downstream tasks. Our study not only offers a valuable user guide but also unveils several new insights. First, if tuned carefully, different PETL methods can obtain similar accuracy in the low-shot benchmark VTAB-1K. This includes simple methods like FT the bias terms that were reported inferior. Second, though with similar accuracy, we find that PETL methods make different mistakes and high-confidence predictions, likely due to their different inductive biases. Such an inconsistency (or complementariness) opens up the opportunity for ensemble methods, and we make preliminary attempts at this. Third, going beyond the commonly used low-shot tasks, we find that PETL is also useful in many-shot regimes -- it achieves comparable and sometimes better accuracy than full FT, using much fewer learnable parameters. Last but not least, we investigate PETL's ability to preserve a pre-trained model's robustness to distribution shifts (e.g., a CLIP backbone). Perhaps not surprisingly, PETL methods outperform full FT alone. However, with weight-space ensembles, the fully fine-tuned model can better balance target (i.e., downstream) distribution and distribution shift performance, suggesting a future research direction for PETL.

VioLET: Vision-Language Efficient Tuning with Collaborative Multi-modal Gradients

VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control

Neural Collapse Anchored Prompt Tuning for Generalizable Vision-Language Models

Towards a Unified View on Visual Parameter-Efficient Transfer Learning

Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach

When Parameter-efficient Tuning Meets General-purpose Vision-language Models

Gradient Projection For Continual Parameter-Efficient Tuning

Lessons Learned from a Unifying Empirical Study of Parameter-Efficient Transfer Learning (PETL) in Visual Recognition

Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learning

VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigation

PVP: Pre-trained Visual Parameter-Efficient Tuning

Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer

Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation

UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory

Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference

Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models

Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models

Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrieval

Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation

M$^2$IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension