Abstract:One significant factor we expect the video representation learning to capture, especially in contrast with the image representation learning, is the object motion. However, we found that in the current mainstream video datasets, some action categories are highly related with the scene where the action happens, making the model tend to degrade to a solution where only the scene information is encoded. For example, a trained model may predict a video as playing football simply because it sees the field, neglecting that the subject is dancing as a cheerleader on the field. This is against our original intention towards the video representation learning and may bring scene bias on a different dataset that can not be ignored. In order to tackle this problem, we propose to decouple the scene and the motion (DSM) with two simple operations, so that the model attention towards the motion information is better paid. Specifically, we construct a positive clip and a negative clip for each video. Compared to the original video, the positive/negative is motion-untouched/broken but scene-broken/untouched by Spatial Local Disturbance and Temporal Local Disturbance. Our objective is to pull the positive closer while pushing the negative farther to the original clip in the latent space. In this way, the impact of the scene is weakened while the temporal sensitivity of the network is further enhanced. We conduct experiments on two tasks with various backbones and different pre-training datasets, and find that our method surpass the SOTA methods with a remarkable 8.1% and 8.8% improvement towards action recognition task on the UCF101 and HMDB51 datasets respectively using the same backbone.

Controllable Augmentations for Video Representation Learning.

Time-Equivariant Contrastive Video Representation Learning

Contrast and Order Representations for Video Self-supervised Learning.

The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning

Video Representation Learning with Graph Contrastive Augmentation

An Improved Inter-intra Contrastive Learning Framework on Self-supervised Video Representation

Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization

Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization

Improve Video Representation with Temporal Adversarial Augmentation

Motion-Focused Contrastive Learning of Video Representations*

Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework

Motion-aware Contrastive Video Representation Learning Via Foreground-background Merging

Audio-Visual Contrastive Learning with Temporal Self-Supervision

Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting

Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion

Motion Sensitive Contrastive Learning for Self-supervised Video Representation

Don't Judge by the Look: Towards Motion Coherent Video Representation

Video Contrastive Learning with Global Context

ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency

Long Short View Feature Decomposition via Contrastive Video Representation Learning

Frequency Selective Augmentation for Video Representation Learning