Abstract:The majority of efforts on improving the performance of few-shot action recognition are dedicated to designing sophisticated temporal alignment algorithms. However, these works all heavily rely on prior knowledge within a pre-trained model. Recently, CLIP (Contrastive Language-Image Pre-Training) has shown significant few-shot learning capability in various downstream tasks. Existing works fine-tune CLIP directly on the novel classes without considering the potential utilization of the adequately labeled base class data. In this work, we conduct a thorough exploration of the adaptation strategies of CLIP for few-shot action recognition. Our findings reveal that despite using a large-scale pre-trained model such as CLIP, it remains necessary to utilize sufficient base class data, if available, to fine-tune the model rather than directly fine-tuning on the novel classes. Moreover, we compare two classical adaptation algorithms proposed to address few-shot learning: Meta-learning and Finetuning \footnoteFor clarity, in this paper we use " Finetuning '' to denote a specific adaptation strategy different from " Meta-learning '', and use "fine-tuning'', which is non-italicized and hyphenated, to denote modifying the parameters of a pre-trained model. Our results indicate that Meta-learning is the better method to inspire the generalization potential of the CLIP. Additionally, we propose to use an overlooked, simple but efficient fine-tuning method: partial fine-tuning, which only fine-tunes the last layer of the backbone. It requires fewer learnable parameters and less computational cost compared to full fine-tuning or fine-tuning additionally introduced adapter modules. Extensive experiments conducted on HMDB51, UCF101, and Kinetics datasets consistently demonstrate the superior generalization ability of our method, which achieves new state-of-the-art results in few-shot action recognition.

Neural Fine-Tuning Search for Few-Shot Learning

Adaptive Fine-Tuning Strategy for Few-Shot Learning

Revisiting Fine-tuning for Few-shot Learning

Efficient Automatic Meta Optimization Search for Few-Shot Learning

Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning

Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learning

Meta Navigator: Search for a Good Adaptation Policy for Few-shot Learning

Meta-Learning Adversarial Domain Adaptation Network for Few-Shot Text Classification.

Hybrid Fine-Tuning Strategy for Few-Shot Classification

Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks

EMAS: Efficient Meta Architecture Search for Few-Shot Learning

Exploring the Adaptation Strategy of CLIP for Few-Shot Action Recognition

Meta-Transfer-Adjustment Learning for Few-Shot Learning

SelectNAdapt: Support Set Selection for Few-Shot Domain Adaptation

Active Few-Shot Fine-Tuning

A Neocognitron Based on Multi-objective Optimization for Few-Shot Learning

Instance-Level Embedding Adaptation for Few-Shot Learning

Meta-learning methodology based on meta-unsupervised algorithm for meta-model selection to solve few-shot base-tasks

Loss Architecture Search for Few-Shot Object Recognition