Abstract:Parameter-efficient transfer learning (PETL) has attracted significant attention lately, due to the increasing size of pre-trained models and the need to fine-tune (FT) them for superior downstream performance. This community-wide enthusiasm has sparked a plethora of approaches. Nevertheless, a systematic study to understand their performance and suitable application scenarios is lacking, leaving questions like when to apply PETL and which approach to use largely unanswered. In this paper, we conduct a unifying empirical study of representative PETL methods in the context of Vision Transformers. We systematically tune their hyper-parameters to fairly compare their accuracy on downstream tasks. Our study not only offers a valuable user guide but also unveils several new insights. First, if tuned carefully, different PETL methods can obtain similar accuracy in the low-shot benchmark VTAB-1K. This includes simple methods like FT the bias terms that were reported inferior. Second, though with similar accuracy, we find that PETL methods make different mistakes and high-confidence predictions, likely due to their different inductive biases. Such an inconsistency (or complementariness) opens up the opportunity for ensemble methods, and we make preliminary attempts at this. Third, going beyond the commonly used low-shot tasks, we find that PETL is also useful in many-shot regimes -- it achieves comparable and sometimes better accuracy than full FT, using much fewer learnable parameters. Last but not least, we investigate PETL's ability to preserve a pre-trained model's robustness to distribution shifts (e.g., a CLIP backbone). Perhaps not surprisingly, PETL methods outperform full FT alone. However, with weight-space ensembles, the fully fine-tuned model can better balance target (i.e., downstream) distribution and distribution shift performance, suggesting a future research direction for PETL.

Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm

One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

Lessons Learned from a Unifying Empirical Study of Parameter-Efficient Transfer Learning (PETL) in Visual Recognition

Rethinking Efficient Tuning Methods from a Unified Perspective

Towards a Unified View on Visual Parameter-Efficient Transfer Learning

Parameter-Efficient Transfer Learning of Audio Spectrogram Transformers

VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigation

Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learning

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models

UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory

Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach

SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning

Towards Optimal Adapter Placement for Efficient Transfer Learning

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

Towards Efficient Visual Adaption via Structural Re-parameterization

Towards a Unified View of Parameter-Efficient Transfer Learning

ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts

PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning

Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets

Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Models

Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrieval