Abstract:Self-supervised representation learning for videos has been very attractive recently because these methods exploit the information inherently obtained from the video itself instead of annotated labels that is quite time-consuming. However, existing methods ignore the importance of global observation while performing spatio-temporal transformation perception, which highly limits the expression capabilities of the video representation. This paper proposes a novel pretext task that combines the temporal information perception of the video with the motion amplitude perception of moving objects to learn the spatio-temporal representation of the video. Specifically, given a video clip containing several video segments, each video segment is sampled by different sampling rates and the order of video segments is disrupted. Then, the network is used to regress the sampling rate of each video segment and classify the order of input video segments. In the pre-training stage, the network can learn rich spatio-temporal semantic information where content-related contrastive learning is introduced to make the learned video representation more discriminative. To alleviate the appearance dependency caused by contrastive learning, we design a novel and robust vector similarity measurement approach, which can take feature alignment into consideration. Moreover, a view synthesis framework is proposed to further improve the performance of contrastive learning by automatically generating reasonable transformed views. We conduct benchmark experiments with several 3D backbone networks on two datasets. The results show that our proposed method outperforms the existing state-of-the-art methods across the three backbones on two downstream tasks of human action recognition and video retrieval.

Temporally-Embedded Self-Supervised Video Representation Learning

Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics

Self-Supervised Spatiotemporal Learning Via Video Clip Order Prediction.

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Self-Supervised Representation Learning for Videos by Segmenting Via Sampling Rate Order Prediction

Temporally Coherent Embeddings for Self-Supervised Video Representation Learning

Self-supervised Temporal Discriminative Learning for Video Representation Learning

Dynamic Spatio-Temporal Feature Learning via Graph Convolution in 3D Convolutional Networks

Spatiotemporal Multimodal Learning With 3D CNNs for Video Action Recognition

Explore Video Clip Order With Self-Supervised and Curriculum Learning for Video Applications

Learning Deep Intrinsic Video Representation by Exploring Temporal Coherence and Graph Structure

Temporal Distinct Representation Learning for Action Recognition

Unsupervised Learning of Long-Term Motion Dynamics for Videos

Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity

Spatial-then-Temporal Self-Supervised Learning for Video Correspondence.

Collaborative Spatio-temporal Feature Learning for Video Action Recognition

Self-Supervised Video Representation Learning with Motion-Contrastive Perception

Temporal Embeddings: Scalable Self-Supervised Temporal Representation Learning from Spatiotemporal Data for Multimodal Computer Vision

Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework

A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning

Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks