Abstract:The emerging field of action prediction plays a vital role in various computer vision applications such as autonomous driving, activity analysis and human-computer interaction. Despite significant advancements, accurately predicting future actions remains a challenging problem due to high dimensionality, complex dynamics and uncertainties inherent in video data. Traditional supervised approaches require large amounts of labelled data, which is expensive and time-consuming to obtain. This paper introduces a novel self-supervised video strategy for enhancing action prediction inspired by DINO (self-distillation with no labels). The Temporal-DINO approach employs two models; a 'student' processing past frames; and a 'teacher' processing both past and future frames, enabling a broader temporal context. During training, the teacher guides the student to learn future context by only observing past frames. The strategy is evaluated on ROAD dataset for the action prediction downstream task using 3D-ResNet, Transformer, and LSTM architectures. The experimental results showcase significant improvements in prediction performance across these architectures, with our method achieving an average enhancement of 9.9% Precision Points (PP), highlighting its effectiveness in enhancing the backbones' capabilities of capturing long-term dependencies. Furthermore, our approach demonstrates efficiency regarding the pretraining dataset size and the number of epochs required. This method overcomes limitations present in other approaches, including considering various backbone architectures, addressing multiple prediction horizons, reducing reliance on hand-crafted augmentations, and streamlining the pretraining process into a single stage. These findings highlight the potential of our approach in diverse video-based tasks such as activity recognition, motion planning, and scene understanding.

Unsupervised Learning for Forecasting Action Representations.

Temporal Constrained Feasible Subspace Learning for Human Pose Forecasting

Imitative Learning for Multi-Person Action Forecasting.

Predictive Learning: Using Future Representation Learning Variantial Autoencoder for Human Action Prediction

Unsupervised video forecasting with flow parsing mechanism of human visual system

FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations

Online Action Detection and Forecast Via Multitask Deep Recurrent Neural Networks

Relational Action Forecasting

Unsupervised Learning of View-invariant Action Representations

Intention-Conditioned Long-Term Human Egocentric Action Forecasting

Weakly Supervised Action Anticipation Without Object Annotations

Human Action Forecasting by Learning Task Grammars

Generative Hybrid Representations for Activity Forecasting With No-Regret Learning

Online Action Detection with Learning Future Representations by Contrastive Learning.

Unsupervised learning for physical interaction through video prediction

Weakly-Supervised Dense Action Anticipation

Action-guided 3D Human Motion Prediction.

Multiscale Video Pretraining for Long-Term Activity Forecasting

Peeking Into the Future: Predicting Future Person Activities and Locations in Videos

TTPP: Temporal Transformer with Progressive Prediction for efficient action anticipation

Temporal DINO: A Self-supervised Video Strategy to Enhance Action Prediction