Abstract:Early Action Prediction (EAP) in videos aims at forecasting the action labels from partially observed videos. It is crucial in various applications, including video surveillance, driverless cars, human-robot interaction, and patient activity monitoring. EAP becomes challenging when visual similarity exists between two actions or when one action appears as a subpart of another, leading to interrelated actions. To address this, we propose a novel approach for the early prediction of visually similar and interrelated actions. Our method involves representing each high-level action as temporal compositions of sub-actions, breaking down complex actions into sequences of smaller, more basic, and distinct "local actions." Furthermore, we construct a dictionary where each original action class serves as a key, with corresponding values representing sequences of possible constituent local actions. The proposed method comprises of two-level classifier namely base classifier and sequence classifier. The base classifier is trained on segmented local action classes using a 3DCNN-based architecture. In a partially observed video, segments are classified using the base classifier to obtain local action labels. The sequence of observed action labels is then input into the sequence classifier, predicting the high-level action class label through TF-IDF-based cosine similarity between the observed sequence and the dictionary classes. We evaluated the effectiveness of our approach using two publicly available datasets, SYSU 3D HOI and MSR Daily Activity. Our method achieved notable accuracy, reaching 82.5% on SYSU 3D HOI and 90% on MSR Daily Activity Dataset, by observing just the first 40 percent of frames.

Human Action Co-occurrence in Lifestyle Vlogs using Graph Link Prediction

Identifying Visible Actions in Lifestyle Vlogs

Improved Actor Relation Graph based Group Activity Recognition

Action Genome: Actions as Composition of Spatio-temporal Scene Graphs

Dynamic Graph Modules for Modeling Object-Object Interactions in Activity Recognition

Human Action Recognition Using Deep Learning Methods.

MovieGraphs: Towards Understanding Human-Centric Situations from Videos

When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs

Visual Knowledge Graph for Human Action Reasoning in Videos

Graphing the Future: Activity and Next Active Object Prediction using Graph-based Activity Representations

Spatio-Temporal Action Graph Networks

Action Recognition by Exploring Data Distribution and Feature Correlation

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

Detection and Validation of Macro-Activities in Human Inertial Signals Using Graph Link Prediction

Towards Tokenized Human Dynamics Representation

A Hybrid Graph Network for Complex Activity Detection in Video

Graph-based Sparse Coding and Embedding for Activity-Based Human Identification

Graph-based temporal action co-localization from an untrimmed video

Enhancing early action prediction in videos through temporal composition of sub-actions

Representing Videos As Discriminative Sub-graphs for Action Recognition*

Learning spatio-temporal co-occurrence correlograms for efficient human action classification