Abstract: Since the release of various large-scale natural language processing (NLP) pre-trained models, parameter efficient transfer learning (PETL) has become a popular paradigm capable of achieving impressive performance on various downstream tasks. PETL aims at making good use of the representation knowledge in the pre-trained large models by fine-tuning a small number of parameters. Recently, it has also attracted increasing attention to developing various PETL techniques for vision tasks. Popular PETL techniques such as Prompt-tuning and Adapter have been proposed for high-level visual downstream tasks such as image classification and video recognition. However, Prefix-tuning remains under-explored for vision tasks. In this work, we intend to adapt large video-based models to downstream tasks with a good parameter-accuracy trade-off. Towards this goal, we propose a framework with a unified view called visual-PETL (V-PETL) to investigate the different aspects affecting the trade-off. Specifically, we analyze the positional importance of trainable parameters and differences between NLP and vision tasks in terms of data structures and pre-training mechanisms while implementing various PETL techniques, especially for the under-explored prefix-tuning technique. Based on a comprehensive understanding of differences between NLP and video data, we propose a new variation of prefix-tuning module called parallel attention (PATT) for video-based downstream tasks. An extensive empirical analysis on two video datasets via different frozen backbones has been carried and the findings show that the proposed PATT can effectively contribute to other PETL techniques. An effective scheme Swin-BAPAT derived from the proposed V-PETL framework achieves significantly better performance than the state-of-the-art AdaptFormer-Swin with slightly more parameters and outperforms full-tuning with far less parameters.

VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

Towards Efficient Visual Adaption via Structural Re-parameterization

Efficient Adaptation of Large Vision Transformer Via Adapter Re-Composing

Efficient Transfer Learning for Video-language Foundation Models

Task-Conditional Adapter for Multi-Task Dense Prediction

Parameter-Efficient Transfer Learning for NLP

Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets

Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models

Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions

Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer.

Towards a Unified View on Visual Parameter-Efficient Transfer Learning

Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision

Efficient Computation Sharing for Multi-Task Visual Scene Understanding

Parameter-Efficient Transfer Learning for Audio-Visual-Language Tasks.

Atten-Adapter: A Unified Attention-Based Adapter for Efficient Tuning

Time-, Memory- and Parameter-Efficient Visual Adaptation

Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks