Abstract:Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods. Code is released at https://github.com/wengzejia1/Open-VCLIP.

Rethinking CLIP-based Video Learners in Cross-Domain Open-Vocabulary Action Recognition

ActionCLIP: Adapting Language-Image Pretrained Models for Video Action Recognition.

ActionCLIP: A New Paradigm for Video Action Recognition

Knowledge-guided Pre-Training and Fine-Tuning: Video Representation Learning for Action Recognition

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data

M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition

Adapting CLIP for Action Recognition via Dual Semantic Supervision and Temporal Prompt Reparameterization

Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model Via Interpolated Weight Optimization

Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognition

Fine-grained Knowledge Graph-driven Video-Language Learning for Action Recognition

Action Recognition Via Fine-Tuned CLIP Model and Temporal Transformer.

Annealing Temporal-Spatial Contrastive Learning for Multi-View Online Action Detection

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment

Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment

GBC: Guided Alignment and Adaptive Boosting CLIP Bridging Vision and Language for Robust Action Recognition

CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning

ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation

Video Domain Incremental Learning for Human Action Recognition in Home Environments

Cross-view Action Recognition by Cross-Domain Learning.

OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning