Abstract:Video self-supervised learning is a challenging task, which requires significant expressive power from the model to leverage rich spatial-temporal knowledge and generate effective supervisory signals from large amounts of unlabeled videos. However, existing methods fail to increase the temporal diversity of unlabeled videos and ignore elaborately modeling multi-scale temporal dependencies in an explicit way. To overcome these limitations, we take advantage of the multi-scale temporal dependencies within videos and proposes a novel video self-supervised learning framework named Temporal Contrastive Graph Learning (TCGL), which jointly models the inter-snippet and intra-snippet temporal dependencies for temporal representation learning with a hybrid graph contrastive learning strategy. Specifically, a Spatial-Temporal Knowledge Discovering (STKD) module is first introduced to extract motion-enhanced spatial-temporal representations from videos based on the frequency domain analysis of discrete cosine transform. To explicitly model multi-scale temporal dependencies of unlabeled videos, our TCGL integrates the prior knowledge about the frame and snippet orders into graph structures, i.e., the intra-/inter- snippet Temporal Contrastive Graphs (TCG). Then, specific contrastive learning modules are designed to maximize the agreement between nodes in different graph views. To generate supervisory signals for unlabeled videos, we introduce an Adaptive Snippet Order Prediction (ASOP) module which leverages the relational knowledge among video snippets to learn the global context representation and recalibrate the channel-wise features adaptively. Experimental results demonstrate the superiority of our TCGL over the state-of-the-art methods on large-scale action recognition and video retrieval <a class="link-external link-http" href="http://benchmarks.The" rel="external noopener nofollow">this http URL</a> code is publicly available at <a class="link-external link-https" href="https://github.com/YangLiu9208/TCGL" rel="external noopener nofollow">this https URL</a>.

Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring.

Mug-STAN: Adapting Image-Language Pretrained Models for General Video Understanding.

Transferring Image-CLIP to Video-Text Retrieval via Temporal Relations

Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning

CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning

Implicit Temporal Modeling with Learnable Alignment for Video Recognition

Adapting CLIP for Action Recognition via Dual Semantic Supervision and Temporal Prompt Reparameterization

Action Recognition Via Fine-Tuned CLIP Model and Temporal Transformer.

TDS-CLIP: Temporal Difference Side Network for Image-to-Video Transfer Learning

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment

Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment

Animating Images to Transfer CLIP for Video-Text Retrieval

M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition

TF-CLIP: Learning Text-free CLIP for Video-based Person Re-Identification

Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval

RCAT: Retentive CLIP Adapter Tuning for Improved Video Recognition

TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations

OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning

TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning