Abstract:Pre-trained vision-language models provide a robust foundation for efficient transfer learning across various downstream tasks. In the field of video action recognition, mainstream approaches often introduce additional parameter modules to capture temporal information. While the increased model capacity brought by these additional parameters helps better fit the video-specific inductive biases, existing methods require learning a large number of parameters and are prone to catastrophic forgetting of the original generalizable knowledge. In this paper, we propose a simple yet effective Multi-modal Spatio-Temporal Adapter (MSTA) to improve the alignment between representations in the text and vision branches, achieving a balance between general knowledge and task-specific knowledge. Furthermore, to mitigate over-fitting and enhance generalizability, we introduce a spatio-temporal description-guided consistency constraint. This constraint involves feeding template inputs (i.e., ``a video of $\{\textbf{cls}\}$'') into the trainable language branch, while LLM-generated spatio-temporal descriptions are input into the pre-trained language branch, enforcing consistency between the outputs of the two branches. This mechanism prevents over-fitting to downstream tasks and improves the distinguishability of the trainable branch within the spatio-temporal semantic space. We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised learning. Compared to many state-of-the-art methods, our MSTA achieves outstanding performance across all evaluations, while using only 2-7\% of the trainable parameters in the original model. Code will be avaliable at <a class="link-external link-https" href="https://github.com/chenhaoxing/ETL4Video" rel="external noopener nofollow">this https URL</a>.

Alignment and Generation Adapter for Efficient Video-text Understanding

Cross-Modal Adapter for Text-Video Retrieval

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Probabilistic Adaptation of Text-to-Video Models

Learning Video-Text Aligned Representations for Video Captioning

Unsupervised Prototype Adapter for Vision-Language Models

Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation

Adaptively Building a Video-language Model for Video Captioning and Retrieval Without Massive Video Pretraining

MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval

Mug-STAN: Adapting Image-Language Pretrained Models for General Video Understanding.

Cap4Video++: Enhancing Video Understanding with Auxiliary Captions

Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension

Video-to-Audio Generation with Hidden Alignment

Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models

CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editing

HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter

Center-enhanced video captioning model with multimodal semantic alignment

Efficient Transfer Learning for Video-language Foundation Models

Adaptive Feature Abstraction for Translating Video to Text

Adaptive semantic guidance network for video captioning