Abstract:Parameter-efficient transfer learning (PETL) has attracted significant attention lately, due to the increasing size of pre-trained models and the need to fine-tune (FT) them for superior downstream performance. This community-wide enthusiasm has sparked a plethora of approaches. Nevertheless, a systematic study to understand their performance and suitable application scenarios is lacking, leaving questions like when to apply PETL and which approach to use largely unanswered. In this paper, we conduct a unifying empirical study of representative PETL methods in the context of Vision Transformers. We systematically tune their hyper-parameters to fairly compare their accuracy on downstream tasks. Our study not only offers a valuable user guide but also unveils several new insights. First, if tuned carefully, different PETL methods can obtain similar accuracy in the low-shot benchmark VTAB-1K. This includes simple methods like FT the bias terms that were reported inferior. Second, though with similar accuracy, we find that PETL methods make different mistakes and high-confidence predictions, likely due to their different inductive biases. Such an inconsistency (or complementariness) opens up the opportunity for ensemble methods, and we make preliminary attempts at this. Third, going beyond the commonly used low-shot tasks, we find that PETL is also useful in many-shot regimes -- it achieves comparable and sometimes better accuracy than full FT, using much fewer learnable parameters. Last but not least, we investigate PETL's ability to preserve a pre-trained model's robustness to distribution shifts (e.g., a CLIP backbone). Perhaps not surprisingly, PETL methods outperform full FT alone. However, with weight-space ensembles, the fully fine-tuned model can better balance target (i.e., downstream) distribution and distribution shift performance, suggesting a future research direction for PETL.

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory

SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

DTL: Disentangled Transfer Learning for Visual Recognition

Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification

FPT+: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification

Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach

One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models

Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning

Hierarchical Side-Tuning for Vision Transformers

ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts

READ: Recurrent Adaptation of Large Transformers

Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm

CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models

ConPET: Continual Parameter-Efficient Tuning for Large Language Models

Rethinking Efficient Tuning Methods from a Unified Perspective

Expanding Sparse Tuning for Low Memory Usage

Lessons Learned from a Unifying Empirical Study of Parameter-Efficient Transfer Learning (PETL) in Visual Recognition