Abstract:Parameter-efficient transfer learning (PETL) has attracted significant attention lately, due to the increasing size of pre-trained models and the need to fine-tune (FT) them for superior downstream performance. This community-wide enthusiasm has sparked a plethora of approaches. Nevertheless, a systematic study to understand their performance and suitable application scenarios is lacking, leaving questions like when to apply PETL and which approach to use largely unanswered. In this paper, we conduct a unifying empirical study of representative PETL methods in the context of Vision Transformers. We systematically tune their hyper-parameters to fairly compare their accuracy on downstream tasks. Our study not only offers a valuable user guide but also unveils several new insights. First, if tuned carefully, different PETL methods can obtain similar accuracy in the low-shot benchmark VTAB-1K. This includes simple methods like FT the bias terms that were reported inferior. Second, though with similar accuracy, we find that PETL methods make different mistakes and high-confidence predictions, likely due to their different inductive biases. Such an inconsistency (or complementariness) opens up the opportunity for ensemble methods, and we make preliminary attempts at this. Third, going beyond the commonly used low-shot tasks, we find that PETL is also useful in many-shot regimes -- it achieves comparable and sometimes better accuracy than full FT, using much fewer learnable parameters. Last but not least, we investigate PETL's ability to preserve a pre-trained model's robustness to distribution shifts (e.g., a CLIP backbone). Perhaps not surprisingly, PETL methods outperform full FT alone. However, with weight-space ensembles, the fully fine-tuned model can better balance target (i.e., downstream) distribution and distribution shift performance, suggesting a future research direction for PETL.

B-Pet: The PET Model with Parameter-Efficient Learning

PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning

FPT+: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification

HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models

ConPET: Continual Parameter-Efficient Tuning for Large Language Models

Arbitrary Few Parameters Are Good Enough for Adapting Large-scale Pre-trained Language Models

SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models

Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification

Sparse Structure Search for Parameter-Efficient Tuning

Pet Image Resolution Uniformity Improvements Using Deep Learning

Parameter Efficient Fine Tuning for Multi-scanner PET to PET Reconstruction

When Parameter-efficient Tuning Meets General-purpose Vision-language Models

Exploring the Impact of Model Scaling on Parameter-Efficient Tuning

VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control

One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

Gradient Projection For Continual Parameter-Efficient Tuning

PetS: A Unified Framework for Parameter-Efficient Transformers Serving

Lessons Learned from a Unifying Empirical Study of Parameter-Efficient Transfer Learning (PETL) in Visual Recognition

Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Models