Abstract:Self-supervised representation learning for videos has been very attractive recently because these methods exploit the information inherently obtained from the video itself instead of annotated labels that is quite time-consuming. However, existing methods ignore the importance of global observation while performing spatio-temporal transformation perception, which highly limits the expression capabilities of the video representation. This paper proposes a novel pretext task that combines the temporal information perception of the video with the motion amplitude perception of moving objects to learn the spatio-temporal representation of the video. Specifically, given a video clip containing several video segments, each video segment is sampled by different sampling rates and the order of video segments is disrupted. Then, the network is used to regress the sampling rate of each video segment and classify the order of input video segments. In the pre-training stage, the network can learn rich spatio-temporal semantic information where content-related contrastive learning is introduced to make the learned video representation more discriminative. To alleviate the appearance dependency caused by contrastive learning, we design a novel and robust vector similarity measurement approach, which can take feature alignment into consideration. Moreover, a view synthesis framework is proposed to further improve the performance of contrastive learning by automatically generating reasonable transformed views. We conduct benchmark experiments with several 3D backbone networks on two datasets. The results show that our proposed method outperforms the existing state-of-the-art methods across the three backbones on two downstream tasks of human action recognition and video retrieval.

Self-supervised Video Representation Learning via Capturing Semantic Changes Indicated by Saccades

Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos

STC: Spatio-Temporal Contrastive Learning for Video Instance Segmentation.

Learning Where to Learn in Cross-View Self-Supervised Learning

Self-Supervised Representation Learning for Videos by Segmenting Via Sampling Rate Order Prediction

Self-supervised visual learning from interactions with objects

Self-supervised learning of video representations from a child's perspective

Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos

Self-Supervised Video Similarity Learning

Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework

Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics

SSAN: Separable Self-Attention Network for Video Representation Learning

Self-Supervised Video Representation Learning with Motion-Contrastive Perception

Temporally-Embedded Self-Supervised Video Representation Learning

Semantic Lens: Instance-Centric Semantic Alignment for Video Super-Resolution

Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective

Video Action Recognition with Attentive Semantic Units

Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems

Enhancing Motion Visual Cues for Self-Supervised Video Representation Learning

Audio-Visual Contrastive Learning with Temporal Self-Supervision