Abstract:The majority of efforts on improving the performance of few-shot action recognition are dedicated to designing sophisticated temporal alignment algorithms. However, these works all heavily rely on prior knowledge within a pre-trained model. Recently, CLIP (Contrastive Language-Image Pre-Training) has shown significant few-shot learning capability in various downstream tasks. Existing works fine-tune CLIP directly on the novel classes without considering the potential utilization of the adequately labeled base class data. In this work, we conduct a thorough exploration of the adaptation strategies of CLIP for few-shot action recognition. Our findings reveal that despite using a large-scale pre-trained model such as CLIP, it remains necessary to utilize sufficient base class data, if available, to fine-tune the model rather than directly fine-tuning on the novel classes. Moreover, we compare two classical adaptation algorithms proposed to address few-shot learning: Meta-learning and Finetuning \footnoteFor clarity, in this paper we use " Finetuning '' to denote a specific adaptation strategy different from " Meta-learning '', and use "fine-tuning'', which is non-italicized and hyphenated, to denote modifying the parameters of a pre-trained model. Our results indicate that Meta-learning is the better method to inspire the generalization potential of the CLIP. Additionally, we propose to use an overlooked, simple but efficient fine-tuning method: partial fine-tuning, which only fine-tunes the last layer of the backbone. It requires fewer learnable parameters and less computational cost compared to full fine-tuning or fine-tuning additionally introduced adapter modules. Extensive experiments conducted on HMDB51, UCF101, and Kinetics datasets consistently demonstrate the superior generalization ability of our method, which achieves new state-of-the-art results in few-shot action recognition.

Adapting CLIP for Action Recognition via Dual Semantic Supervision and Temporal Prompt Reparameterization

ActionCLIP: Adapting Language-Image Pretrained Models for Video Action Recognition.

M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition

Action Recognition Via Fine-Tuned CLIP Model and Temporal Transformer.

ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization

Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval

ActionCLIP: A New Paradigm for Video Action Recognition

Enhancing Few-Shot CLIP With Semantic-Aware Fine-Tuning

Rethinking Visual Content Refinement in Low-Shot CLIP Adaptation

Rethinking CLIP-based Video Learners in Cross-Domain Open-Vocabulary Action Recognition

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data

FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

PromptLearner-CLIP: Contrastive Multi-Modal Action Representation Learning with Context Optimization

ActPrompt: In-Domain Feature Adaptation via Action Cues for Video Temporal Grounding

MA-FSAR: Multimodal Adaptation of CLIP for Few-Shot Action Recognition

CLIP-SP: Vision-language Model with Adaptive Prompting for Scene Parsing

Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring.

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment

Exploring the Adaptation Strategy of CLIP for Few-Shot Action Recognition