Abstract:Pre-trained vision-language models provide a robust foundation for efficient transfer learning across various downstream tasks. In the field of video action recognition, mainstream approaches often introduce additional parameter modules to capture temporal information. While the increased model capacity brought by these additional parameters helps better fit the video-specific inductive biases, existing methods require learning a large number of parameters and are prone to catastrophic forgetting of the original generalizable knowledge. In this paper, we propose a simple yet effective Multi-modal Spatio-Temporal Adapter (MSTA) to improve the alignment between representations in the text and vision branches, achieving a balance between general knowledge and task-specific knowledge. Furthermore, to mitigate over-fitting and enhance generalizability, we introduce a spatio-temporal description-guided consistency constraint. This constraint involves feeding template inputs (i.e., ``a video of $\{\textbf{cls}\}$'') into the trainable language branch, while LLM-generated spatio-temporal descriptions are input into the pre-trained language branch, enforcing consistency between the outputs of the two branches. This mechanism prevents over-fitting to downstream tasks and improves the distinguishability of the trainable branch within the spatio-temporal semantic space. We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised learning. Compared to many state-of-the-art methods, our MSTA achieves outstanding performance across all evaluations, while using only 2-7\% of the trainable parameters in the original model. Code will be avaliable at <a class="link-external link-https" href="https://github.com/chenhaoxing/ETL4Video" rel="external noopener nofollow">this https URL</a>.

READ: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Efficient Transfer Learning for Video-language Foundation Models

Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models

ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning

READ: Recurrent Adaptation of Large Transformers

Parameter-Efficient Transfer Learning for Audio-Visual-Language Tasks.

Cross-Modal Adapter for Text-Video Retrieval

MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Alignment and Generation Adapter for Efficient Video-text Understanding

BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

Probabilistic Adaptation of Text-to-Video Models

Unsupervised Prototype Adapter for Vision-Language Models

Exploiting Adapters for Cross-Lingual Low-Resource Speech Recognition

Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning

Towards Efficient Visual Adaption via Structural Re-parameterization

Exploring Adapter-based Transfer Learning for Recommender Systems: Empirical Studies and Practical Insights.

Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models