Abstract:Video self-supervised learning is a challenging task, which requires significant expressive power from the model to leverage rich spatial-temporal knowledge and generate effective supervisory signals from large amounts of unlabeled videos. However, existing methods fail to increase the temporal diversity of unlabeled videos and ignore elaborately modeling multi-scale temporal dependencies in an explicit way. To overcome these limitations, we take advantage of the multi-scale temporal dependencies within videos and proposes a novel video self-supervised learning framework named Temporal Contrastive Graph Learning (TCGL), which jointly models the inter-snippet and intra-snippet temporal dependencies for temporal representation learning with a hybrid graph contrastive learning strategy. Specifically, a Spatial-Temporal Knowledge Discovering (STKD) module is first introduced to extract motion-enhanced spatial-temporal representations from videos based on the frequency domain analysis of discrete cosine transform. To explicitly model multi-scale temporal dependencies of unlabeled videos, our TCGL integrates the prior knowledge about the frame and snippet orders into graph structures, i.e., the intra-/inter- snippet Temporal Contrastive Graphs (TCG). Then, specific contrastive learning modules are designed to maximize the agreement between nodes in different graph views. To generate supervisory signals for unlabeled videos, we introduce an Adaptive Snippet Order Prediction (ASOP) module which leverages the relational knowledge among video snippets to learn the global context representation and recalibrate the channel-wise features adaptively. Experimental results demonstrate the superiority of our TCGL over the state-of-the-art methods on large-scale action recognition and video retrieval <a class="link-external link-http" href="http://benchmarks.The" rel="external noopener nofollow">this http URL</a> code is publicly available at <a class="link-external link-https" href="https://github.com/YangLiu9208/TCGL" rel="external noopener nofollow">this https URL</a>.

Text-Guided Graph Temporal Modeling for Few-Shot Video Classification

Exploiting Semantic-Level Affinities with a Mask-Guided Network for Temporal Action Proposal in Videos.

Revisiting the Spatial and Temporal Modeling for Few-shot Action Recognition

Few-Shot Video Classification via Temporal Alignment

Exploring cross-video matching for few-shot video classification via dual-hierarchy graph neural network learning

Few-Shot Temporal Sentence Grounding Via Memory-Guided Semantic Learning

Boosting Few-shot Action Recognition with Graph-guided Hybrid Matching

TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning

Temporal Alignment Prediction for Few-Shot Video Classification

Temporal Multimodal Graph Transformer With Global-Local Alignment for Video-Text Retrieval

Training-free Video Temporal Grounding using Large-scale Pre-trained Models

Improving Few-shot Text Classification via Pretrained Language Representations

Learning Implicit Temporal Alignment for Few-shot Video Classification

Multi-grained Temporal Prototype Learning for Few-shot Video Object Segmentation

Graph Complemented Latent Representation for Few-Shot Image Classification

Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification

Task-specific alignment and multiple-level transformer for few-shot action recognition

Semantic-guided spatio-temporal attention for few-shot action recognition

Multi-Modal Few-Shot Temporal Action Detection

Few-shot Fuzzy Temporal Knowledge Graph Completion Via Fuzzy Semantics and Dynamic Attention Network