Abstract:Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods. Code is released at https://github.com/wengzejia1/Open-VCLIP.

Unleash the Potential of CLIP for Video Highlight Detection

SegCLIP: Multimodal Visual-Language and Prompt Learning for High-Resolution Remote Sensing Semantic Segmentation

Delving into CLIP latent space for Video Anomaly Recognition

LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models

Unsupervised Modality-Transferable Video Highlight Detection With Representation Activation Sequence Learning

VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval

How Much Can CLIP Benefit Vision-and-Language Tasks?

DiffCLIP: Few-shot Language-driven Multimodal Classifier

ChangeCLIP: Remote sensing change detection with multimodal vision-language representation learning

Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss

CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation

TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection

LLM2CLIP: Powerful Language Model Unlocks Richer Visual Representation

From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data

Cross-category Video Highlight Detection Via Set-based Learning

VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending

Collaborative Noisy Label Cleaner: Learning Scene-aware Trailers for Multi-modal Highlight Detection in Movies

OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning