Abstract:Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods. Code is released at https://github.com/wengzejia1/Open-VCLIP.

XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding

How Much Can CLIP Benefit Vision-and-Language Tasks?

CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance

Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data

LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models

SpaceCLIP: A Vision-Language Pretraining Framework With Spatial Reconstruction On Text

CLIPVQA:Video Quality Assessment via CLIP

Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models

Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training

MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization

Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification

UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding

Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation