Abstract:The majority of efforts on improving the performance of few-shot action recognition are dedicated to designing sophisticated temporal alignment algorithms. However, these works all heavily rely on prior knowledge within a pre-trained model. Recently, CLIP (Contrastive Language-Image Pre-Training) has shown significant few-shot learning capability in various downstream tasks. Existing works fine-tune CLIP directly on the novel classes without considering the potential utilization of the adequately labeled base class data. In this work, we conduct a thorough exploration of the adaptation strategies of CLIP for few-shot action recognition. Our findings reveal that despite using a large-scale pre-trained model such as CLIP, it remains necessary to utilize sufficient base class data, if available, to fine-tune the model rather than directly fine-tuning on the novel classes. Moreover, we compare two classical adaptation algorithms proposed to address few-shot learning: Meta-learning and Finetuning \footnoteFor clarity, in this paper we use " Finetuning '' to denote a specific adaptation strategy different from " Meta-learning '', and use "fine-tuning'', which is non-italicized and hyphenated, to denote modifying the parameters of a pre-trained model. Our results indicate that Meta-learning is the better method to inspire the generalization potential of the CLIP. Additionally, we propose to use an overlooked, simple but efficient fine-tuning method: partial fine-tuning, which only fine-tunes the last layer of the backbone. It requires fewer learnable parameters and less computational cost compared to full fine-tuning or fine-tuning additionally introduced adapter modules. Extensive experiments conducted on HMDB51, UCF101, and Kinetics datasets consistently demonstrate the superior generalization ability of our method, which achieves new state-of-the-art results in few-shot action recognition.

Action Recognition Via Fine-Tuned CLIP Model and Temporal Transformer.

ActionCLIP: Adapting Language-Image Pretrained Models for Video Action Recognition.

Knowledge-guided Pre-Training and Fine-Tuning: Video Representation Learning for Action Recognition

Revisiting the Spatial and Temporal Modeling for Few-shot Action Recognition

Adapting CLIP for Action Recognition via Dual Semantic Supervision and Temporal Prompt Reparameterization

Temporal Modeling Approach for Video Action Recognition Based on Vision-language Models.

ActionCLIP: A New Paradigm for Video Action Recognition

M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition

Leveraging Temporal Contextualization for Video Action Recognition

Temporal Distinct Representation Learning for Action Recognition

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data

Transferring Image-CLIP to Video-Text Retrieval via Temporal Relations

Temporal Transformer Networks with Self-Supervision for Action Recognition.

Action recognition with temporal scale-invariant deep learning framework

Exploring the Adaptation Strategy of CLIP for Few-Shot Action Recognition

Long-Short Temporal Contrastive Learning of Video Transformers

Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP

Rethinking CLIP-based Video Learners in Cross-Domain Open-Vocabulary Action Recognition

Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring.

Co-training Transformer with Videos and Images Improves Action Recognition

When Spatial meets Temporal in Action Recognition