Abstract:Self-supervised representation learning for videos has been very attractive recently because these methods exploit the information inherently obtained from the video itself instead of annotated labels that is quite time-consuming. However, existing methods ignore the importance of global observation while performing spatio-temporal transformation perception, which highly limits the expression capabilities of the video representation. This paper proposes a novel pretext task that combines the temporal information perception of the video with the motion amplitude perception of moving objects to learn the spatio-temporal representation of the video. Specifically, given a video clip containing several video segments, each video segment is sampled by different sampling rates and the order of video segments is disrupted. Then, the network is used to regress the sampling rate of each video segment and classify the order of input video segments. In the pre-training stage, the network can learn rich spatio-temporal semantic information where content-related contrastive learning is introduced to make the learned video representation more discriminative. To alleviate the appearance dependency caused by contrastive learning, we design a novel and robust vector similarity measurement approach, which can take feature alignment into consideration. Moreover, a view synthesis framework is proposed to further improve the performance of contrastive learning by automatically generating reasonable transformed views. We conduct benchmark experiments with several 3D backbone networks on two datasets. The results show that our proposed method outperforms the existing state-of-the-art methods across the three backbones on two downstream tasks of human action recognition and video retrieval.

Self-Supervised Spatiotemporal Learning Via Video Clip Order Prediction.

Explore Video Clip Order With Self-Supervised and Curriculum Learning for Video Applications

Skip-Clip: Self-Supervised Spatiotemporal Representation Learning by Future Clip Order Ranking

Temporally-Embedded Self-Supervised Video Representation Learning

Self-Supervised Representation Learning for Videos by Segmenting Via Sampling Rate Order Prediction

Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videos

Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics

Self-Supervised Visual Learning by Variable Playback Speeds Prediction of a Video

Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning

Self-supervised Temporal Discriminative Learning for Video Representation Learning

Spatial-then-Temporal Self-Supervised Learning for Video Correspondence.

Self-labeling video prediction

A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning

Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity

Patch Spatio-Temporal Relation Prediction for Video Anomaly Detection

ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning

Spatial-temporal 3D dependency matching with self-supervised deep learning for monocular visual sensing

Attentive spatial-temporal contrastive learning for self-supervised video representation

Self-Supervised Visual Planning with Temporal Skip Connections

Video Playback Rate Perception For Self-Supervised Spatio-Temporal Representation Learning

Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning