Abstract:Video recognition models often learn scene-biased action representation due to the spurious correlation between actions and scenes in the training data. Such models show poor performance when the test data consists of videos with unseen action-scene combinations. Although scene-debiased action recognition models might address the issue, they often overlook valuable scene information in the data. To address this challenge, we propose to learn DisEntangled VIdeo representations of Action and Scene (DEVIAS), for more holistic video understanding. We propose an encoder-decoder architecture to learn disentangled action and scene representations with a single model. The architecture consists of a disentangling encoder (DE), an action mask decoder (AMD), and a prediction head. The key to achieving the disentanglement is employing both DE and AMD during training time. The DE uses the slot attention mechanism to learn disentangled action and scene representations. For further disentanglement, an AMD learns to predict action masks, given an action slot. With the resulting disentangled representations, we can achieve robust performance across diverse scenarios, including both seen and unseen action-scene combinations. We rigorously validate the proposed method on the UCF-101, Kinetics-400, and HVU datasets for the seen, and the SCUBA, HAT, and HVU datasets for unseen action-scene combination scenarios. Furthermore, DEVIAS provides flexibility to adjust the emphasis on action or scene information depending on dataset characteristics for downstream tasks. DEVIAS shows favorable performance in various downstream tasks: Diving48, Something-Something-V2, UCF-101, and ActivityNet. The code is available at <a class="link-external link-https" href="https://github.com/KHU-VLL/DEVIAS" rel="external noopener nofollow">this https URL</a>.

Learning Discriminative Representations in Videos via Active Embedding Distance Correlation

Temporal Distinct Representation Learning for Action Recognition

Learning Discriminative Spatio-temporal Representations for Semi-supervised Action Recognition

Collaborative Spatio-temporal Feature Learning for Video Action Recognition

Discriminative Segment Focus Network for Fine-grained Video Action Recognition

Beyond Euclidean: Dual-Space Representation Learning for Weakly Supervised Video Violence Detection

Self-supervised Temporal Discriminative Learning for Video Representation Learning

Spatiotemporal Multimodal Learning With 3D CNNs for Video Action Recognition

Memory-augmented Dense Predictive Coding for Video Representation Learning

Learning Spatiotemporal Features via Video and Text Pair Discrimination

Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion

DEVIAS: Learning Disentangled Video Representations of Action and Scene

Hypercorrelation Evolution for Video Class-Incremental Learning

Discriminative Feature Learning with Constraints of Category and Temporal for Action Recognition.

Dual Contrastive Learning for Spatio-temporal Representation

Learning Attention-Enhanced Spatiotemporal Representation for Action Recognition

Cooperative Cross-Stream Network for Discriminative Action Representation

Learning Comprehensive Motion Representation for Action Recognition

Multi-Task Learning of Generalizable Representations for Video Action Recognition

Learning Task-Specific Representation for Video Anomaly Detection with Spatial-Temporal Attention

Temporally-Embedded Self-Supervised Video Representation Learning