Abstract:Many existing methods formulate the action prediction task as recognizing early parts of actions in trimmed videos. In this paper, we focus on predicting actions from ongoing untrimmed videos where actions might not happen at the very beginning of videos. It is extremely challenging to predict actions in such untrimmed videos due to ambiguous or even no information of actions in the early parts of videos. To address this problem, we propose a prediction confidence that assesses the decision quality of a prediction model. Guided by the confidence, the model continuously refines the prediction results by itself with the increasing observed video frames. Specifically, we build a Self Prediction Refining Network (SPR-Net) which incrementally learns the confidence for action prediction. SPR-Net consists of three modules: a temporal hybrid network, an incremental confidence learner, and a self-refining Gumbel softmax sampler. The temporal hybrid network generates the action category distributions by integrating static scene and dynamic motion information. The incremental confidence learner calculates the confidence in an incremental manner, judging the extent to which the temporal hybrid network should believe its prediction result. The self-refining Gumbel softmax sampler models the mutual relationship between the prediction confidence and the category distribution, which enables them to be jointly learned in an end-to-end fashion. We also present a sparse self-attention mechanism to encode local spatio-temporal features into the frame-level motion representation to further improve the prediction performance. Extensive experiments on five datasets (i.e., UT-Interaction, BIT-Interaction, UCF101, THUMOS14, and ActivityNet) validate the effectiveness of the proposed method.

A Discussion of Data Sampling Strategies for Early Action Prediction

Action Recognition Based on A Selective Sampling Strategy for Real-Time Video Surveillance

Spatial–Temporal Context-Aware Online Action Detection and Prediction

A Closer Look at Video Sampling for Sequential Action Recognition

Dynamic Sampling Networks for Efficient Action Recognition in Videos.

Feature Sampling Strategies for Action Recognition

Human Action Recognition with Optimized Video Densely Sampling.

Progressive Teacher-Student Learning For Early Action Prediction

LAP-Net: Adaptive Features Sampling via Learning Action Progression for Online Action Detection

Action Recognition by Saliency-Based Dense Sampling

Early Action Prediction with Generative Adversarial Networks

Rich Action-semantic Consistent Knowledge for Early Action Prediction

Action Density Based Frame Sampling for Human Action Recognition in Videos

Task-adaptive Spatial-Temporal Video Sampler for Few-shot Action Recognition

Finding Action Tubes with a Sparse-to-Dense Framework

MGSampler: an Explainable Sampling Strategy for Video Action Recognition

Action Recognition by Exploring Data Distribution and Feature Correlation

Action Recognition Based on Divide-and-Conquer

Enhancing early action prediction in videos through temporal composition of sub-actions

Confidence-Guided Self Refinement for Action Prediction in Untrimmed Videos