Abstract:Video self-supervised learning is a challenging task, which requires significant expressive power from the model to leverage rich spatial-temporal knowledge and generate effective supervisory signals from large amounts of unlabeled videos. However, existing methods fail to increase the temporal diversity of unlabeled videos and ignore elaborately modeling multi-scale temporal dependencies in an explicit way. To overcome these limitations, we take advantage of the multi-scale temporal dependencies within videos and proposes a novel video self-supervised learning framework named Temporal Contrastive Graph Learning (TCGL), which jointly models the inter-snippet and intra-snippet temporal dependencies for temporal representation learning with a hybrid graph contrastive learning strategy. Specifically, a Spatial-Temporal Knowledge Discovering (STKD) module is first introduced to extract motion-enhanced spatial-temporal representations from videos based on the frequency domain analysis of discrete cosine transform. To explicitly model multi-scale temporal dependencies of unlabeled videos, our TCGL integrates the prior knowledge about the frame and snippet orders into graph structures, i.e., the intra-/inter- snippet Temporal Contrastive Graphs (TCG). Then, specific contrastive learning modules are designed to maximize the agreement between nodes in different graph views. To generate supervisory signals for unlabeled videos, we introduce an Adaptive Snippet Order Prediction (ASOP) module which leverages the relational knowledge among video snippets to learn the global context representation and recalibrate the channel-wise features adaptively. Experimental results demonstrate the superiority of our TCGL over the state-of-the-art methods on large-scale action recognition and video retrieval <a class="link-external link-http" href="http://benchmarks.The" rel="external noopener nofollow">this http URL</a> code is publicly available at <a class="link-external link-https" href="https://github.com/YangLiu9208/TCGL" rel="external noopener nofollow">this https URL</a>.

Temporally Consistent Gaussian Random Field For Video Semantic Analysis

Video Annotation Based on Temporally Consistent Gaussian Random Field

Adaptive Video Segmentation Algorithm Using Hidden Conditional Random Fields

Adaptive Foreground and Shadow Segmentation Using Hidden Conditional Random Fields

Deep Spatio-Temporal Random Fields for Efficient Video Segmentation

A General Framework of Video Segmentation to Logical Unit Based on Conditional Random Fields

Video Generalized Semantic Segmentation Via Non-Salient Feature Reasoning and Consistency

Non-parametric Contextual Relationship Learning for Semantic Video Object Segmentation

Exploiting Semantic And Visual Context For Effective Video Annotation

Learning Local and Global Temporal Contexts for Video Semantic Segmentation

Coarse-to-Fine Spatial-Temporal Relationship Inference for Temporal Sentence Grounding.

Video-to-Video Translation with Global Temporal Consistency.

Video-based Spatio-Temporal Scene Graph Generation with Efficient Self-Supervision Tasks

Temporal Relational Modeling with Self-Supervision for Action Segmentation

TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning

Improving Video Concept Detection Using Spatio-Temporal Correlation

Temporal Contrastive Graph Learning for Video Action Recognition and Retrieval

Video Captioning With Temporal And Region Graph Convolution Network

Conditional Video Diffusion Network for Fine-grained Temporal Sentence Grounding

Multi-class Video Objects Segmentation Based on Conditional Random Fields