Abstract:Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods. Code is released at https://github.com/wengzejia1/Open-VCLIP.

Videoclip: Contrastive pre-training for zero-shot video-text understanding

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data

Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model Via Interpolated Weight Optimization

ProtoCLIP: Prototypical Contrastive Language Image Pretraining

UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding

Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP

CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning

Non-Contrastive Learning Meets Language-Image Pre-Training

X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval

Understanding Chinese Video and Language Via Contrastive Multimodal Pre-Training

MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment

Masked Contrastive Pre-Training for Efficient Video-Text Retrieval

Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting

RankCLIP: Ranking-Consistent Language-Image Pretraining

VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models

Contrastive Language-Action Pre-training for Temporal Localization