Abstract:Video predictive understanding encompasses a wide range of efforts that are concerned with the anticipation of the unobserved future from the current as well as historical video observations. Action prediction is a major sub-area of video predictive understanding and is the focus of this review. This sub-area has two major subdivisions: early action recognition and future action prediction. Early action recognition is concerned with recognizing an ongoing action as soon as possible. Future action prediction is concerned with the anticipation of actions that follow those previously observed. In either case, the \textbf{\textit{causal}} relationship between the past, current, and potential future information is the main focus. Various mathematical tools such as Markov Chains, Gaussian Processes, Auto-Regressive modeling, and Bayesian recursive filtering are widely adopted jointly with computer vision techniques for these two tasks. However, these approaches face challenges such as the curse of dimensionality, poor generalization, and constraints from domain-specific knowledge. Recently, structures that rely on deep convolutional neural networks and recurrent neural networks have been extensively proposed for improving the performance of existing vision tasks, in general, and action prediction tasks, in particular. However, they have their own shortcomings, \eg reliance on massive training data and lack of strong theoretical underpinnings. In this survey, we start by introducing the major sub-areas of the broad area of video predictive understanding, which recently have received intensive attention and proven to have practical value. Next, a thorough review of various early action recognition and future action prediction algorithms are provided with suitably organized divisions. Finally, we conclude our discussion with future research directions.

Online human action detection and anticipation in videos: A survey

Human Action Anticipation: A Survey

A Survey on Deep Learning Techniques for Action Anticipation

Advances in Human Action Recognition: A Survey

Exploring 3d Human Action Recognition: From Offline To Online

A Survey of Human Action Recognition and Posture Prediction

Deep Learning-Based Action Detection in Untrimmed Videos: A Survey

Current Advances on Deep Learning-based Human Action Recognition from Videos: a Survey

Spatial–Temporal Context-Aware Online Action Detection and Prediction

View-invariant action recognition:a survey

Survey on vision-based human action understanding

Human Behavior Analysis: A Survey on Action Recognition

A Survey of Video-based Action Quality Assessment

Action Recognition by Exploring Data Distribution and Feature Correlation

Human Action Recognition: A Taxonomy-Based Survey, Updates, and Opportunities

Study of human action representation in video sequences

A Comprehensive Survey of Vision-Based Human Action Recognition Methods

Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

A survey on deep learning-based spatio-temporal action detection

A Survey on Temporal Action Localization

Human Action Recognition Using Deep Learning Methods.