Abstract:Most existing self-supervised works learn video representation by using a single pretext task. A single pretext task, providing single supervision from unlabeled data, may neglect to describe the difference between spatial features and temporal features. The similar spatial features and temporal features may hinder distinguishing between two similar videos with different class labels. In this paper, we propose an attentive spatial–temporal contrastive learning network (ASTCNet), which learns self-attention spatial–temporal features by contrastive learning between multiple spatial and temporal pretext tasks. The spatial features are learned by multiple spatial pretext tasks, including spatial rotation, and spatial jigsaw. Each spatial feature is enhanced with spatial self-attention by learning the relation between patches. The temporal features are learned by multiple temporal pretext tasks, including temporal order, and temporal pace. Each temporal feature is enhanced with temporal self-attention by learning the relation between frames, and is enhanced by feeding the optical flow features into a motion module. To separate the spatial feature and the temporal feature learned in one video, we represent the video as different features for each pretext task, and design pretext task-based contrastive loss. The pretext task-based contrastive loss encourages the different pretext tasks to learn dissimilar features, and encourages the same pretext task to learn similar features. The pretext task-based contrastive loss can learn the discriminative features for each pretext task in one video. The experiments show that our method achieves state-of-the-art performance for self-supervised action recognition on the UCF101 dataset and the HMDB51 dataset.

Self-supervised pretext task collaborative multi-view contrastive learning for video action recognition

Knowledge-guided Pre-Training and Fine-Tuning: Video Representation Learning for Action Recognition

Attentive spatial-temporal contrastive learning for self-supervised video representation

Cross-view motion consistent self-supervised video inter-intra contrastive for action representation understanding

Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos

Self-Supervised Video Representation Learning with Motion-Contrastive Perception

Collaboratively Self-supervised Video Representation Learning for Action Recognition

Contrast and Order Representations for Video Self-supervised Learning.

Multi-Task Learning of Generalizable Representations for Video Action Recognition

Self-supervised and Weakly Supervised Contrastive Learning for Frame-wise Action Representations

Language-based Action Concept Spaces Improve Video Self-Supervised Learning

Motion-aware Contrastive Video Representation Learning Via Foreground-background Merging

Self-Supervised Video Representation Learning with Meta-Contrastive Network

Cross-Architecture Self-supervised Video Representation Learning

Motion Sensitive Contrastive Learning for Self-supervised Video Representation

Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework

Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting

Learning Transferable Self-attentive Representations for Action Recognition in Untrimmed Videos with Weak Supervision

Audio-Visual Contrastive Learning with Temporal Self-Supervision

Temporal Contrastive Graph Learning for Video Action Recognition and Retrieval

Learning Discriminative Spatio-temporal Representations for Semi-supervised Action Recognition