Abstract:Parameter-efficient transfer learning (PETL) has attracted significant attention lately, due to the increasing size of pre-trained models and the need to fine-tune (FT) them for superior downstream performance. This community-wide enthusiasm has sparked a plethora of approaches. Nevertheless, a systematic study to understand their performance and suitable application scenarios is lacking, leaving questions like when to apply PETL and which approach to use largely unanswered. In this paper, we conduct a unifying empirical study of representative PETL methods in the context of Vision Transformers. We systematically tune their hyper-parameters to fairly compare their accuracy on downstream tasks. Our study not only offers a valuable user guide but also unveils several new insights. First, if tuned carefully, different PETL methods can obtain similar accuracy in the low-shot benchmark VTAB-1K. This includes simple methods like FT the bias terms that were reported inferior. Second, though with similar accuracy, we find that PETL methods make different mistakes and high-confidence predictions, likely due to their different inductive biases. Such an inconsistency (or complementariness) opens up the opportunity for ensemble methods, and we make preliminary attempts at this. Third, going beyond the commonly used low-shot tasks, we find that PETL is also useful in many-shot regimes -- it achieves comparable and sometimes better accuracy than full FT, using much fewer learnable parameters. Last but not least, we investigate PETL's ability to preserve a pre-trained model's robustness to distribution shifts (e.g., a CLIP backbone). Perhaps not surprisingly, PETL methods outperform full FT alone. However, with weight-space ensembles, the fully fine-tuned model can better balance target (i.e., downstream) distribution and distribution shift performance, suggesting a future research direction for PETL.

Parameter-Efficient Transfer Learning of Audio Spectrogram Transformers

Adapter Incremental Continual Learning of Efficient Audio Spectrogram Transformers

Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets

Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptation

Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm

Parameter-Efficient Transfer Learning for Music Foundation Models

Parameter-Efficient Transfer Learning for NLP

Towards a Unified View on Visual Parameter-Efficient Transfer Learning

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models

Lessons Learned from a Unifying Empirical Study of Parameter-Efficient Transfer Learning (PETL) in Visual Recognition

Rethinking Efficient Tuning Methods from a Unified Perspective

One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

Towards Optimal Adapter Placement for Efficient Transfer Learning

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

Parameter-Efficient Fine-Tuning With Adapters

PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context

Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters

AAT: Adapting Audio Transformer for Various Acoustics Recognition Tasks

VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control