Abstract: Since the release of various large-scale natural language processing (NLP) pre-trained models, parameter efficient transfer learning (PETL) has become a popular paradigm capable of achieving impressive performance on various downstream tasks. PETL aims at making good use of the representation knowledge in the pre-trained large models by fine-tuning a small number of parameters. Recently, it has also attracted increasing attention to developing various PETL techniques for vision tasks. Popular PETL techniques such as Prompt-tuning and Adapter have been proposed for high-level visual downstream tasks such as image classification and video recognition. However, Prefix-tuning remains under-explored for vision tasks. In this work, we intend to adapt large video-based models to downstream tasks with a good parameter-accuracy trade-off. Towards this goal, we propose a framework with a unified view called visual-PETL (V-PETL) to investigate the different aspects affecting the trade-off. Specifically, we analyze the positional importance of trainable parameters and differences between NLP and vision tasks in terms of data structures and pre-training mechanisms while implementing various PETL techniques, especially for the under-explored prefix-tuning technique. Based on a comprehensive understanding of differences between NLP and video data, we propose a new variation of prefix-tuning module called parallel attention (PATT) for video-based downstream tasks. An extensive empirical analysis on two video datasets via different frozen backbones has been carried and the findings show that the proposed PATT can effectively contribute to other PETL techniques. An effective scheme Swin-BAPAT derived from the proposed V-PETL framework achieves significantly better performance than the state-of-the-art AdaptFormer-Swin with slightly more parameters and outperforms full-tuning with far less parameters.

Towards Optimal Adapter Placement for Efficient Transfer Learning

Towards Efficient Visual Adaption via Structural Re-parameterization

Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets

Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm

Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy

Adaptable Adapters

p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models

Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions

G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition

Parameter-Efficient Transfer Learning for NLP

Parameter-Efficient Fine-Tuning With Adapters

Parameter-Efficient Transfer Learning of Audio Spectrogram Transformers

One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning

Parameter-Efficient Sparse Retrievers and Rerankers using Adapters

Gradient Projection For Continual Parameter-Efficient Tuning

ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning

Towards a Unified View on Visual Parameter-Efficient Transfer Learning

Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision

UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory