Enhancing early action prediction in videos through temporal composition of sub-actions
Ashwini S. Gavali,Sangeeta.N. Kakarwal
DOI: https://doi.org/10.1007/s11042-024-18870-0
IF: 2.577
2024-03-19
Multimedia Tools and Applications
Abstract:Early Action Prediction (EAP) in videos aims at forecasting the action labels from partially observed videos. It is crucial in various applications, including video surveillance, driverless cars, human-robot interaction, and patient activity monitoring. EAP becomes challenging when visual similarity exists between two actions or when one action appears as a subpart of another, leading to interrelated actions. To address this, we propose a novel approach for the early prediction of visually similar and interrelated actions. Our method involves representing each high-level action as temporal compositions of sub-actions, breaking down complex actions into sequences of smaller, more basic, and distinct "local actions." Furthermore, we construct a dictionary where each original action class serves as a key, with corresponding values representing sequences of possible constituent local actions. The proposed method comprises of two-level classifier namely base classifier and sequence classifier. The base classifier is trained on segmented local action classes using a 3DCNN-based architecture. In a partially observed video, segments are classified using the base classifier to obtain local action labels. The sequence of observed action labels is then input into the sequence classifier, predicting the high-level action class label through TF-IDF-based cosine similarity between the observed sequence and the dictionary classes. We evaluated the effectiveness of our approach using two publicly available datasets, SYSU 3D HOI and MSR Daily Activity. Our method achieved notable accuracy, reaching 82.5% on SYSU 3D HOI and 90% on MSR Daily Activity Dataset, by observing just the first 40 percent of frames.
computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering