Abstract:Existing action detection algorithms usually generate action proposals through an extensive search over the video at multiple temporal scales, which brings about huge computational overhead and deviates from the human perception procedure. We argue that the process of detecting actions in the video should be naturally one of observation and refinement: observe the current temporal window and refine the span of attended window to cover true action regions. In this paper, we propose an active action detection model that learns to search actions through continuously adjusting the bounds of temporal attended window in a self-adaptive way. The whole process can be deemed as an exploring procedure, where an agent is first placed at the beginning of the video and then traverses the whole video by adopting a sequence of transformations on the current attended window to discover actions according to a learned policy. We utilize reinforcement learning, especially the deep Q-learning algorithm to learn the agent's decision policy. Actually, we construct an end-to-end trainable framework for the action detection task, which includes a proposal generation network based on deep Q-learning, and the classification and regression networks responsible for the action category prediction and the action location adjustment, respectively. In addition, we design a long short-term memory structure upon extracted convolutional neural network features of sparsely sampled frames to generate the effective feature representations for video sequences of various durations. We evaluate the action proposal performance of our approach on THUMOS'14 and assess the generalization ability for unseen action categories on ActivityNet. We also compare the action detection performance of ours with other state-of-the-art methods on both data sets. Experiment results validate the effectiveness of the proposed approach, which can achieve comparative or superior performance than other action detection methods via much fewer proposals.

Online Action Detection and Forecast Via Multitask Deep Recurrent Neural Networks

Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networks

Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection

Graph-Temporal LSTM Networks for Skeleton-Based Action Recognition

Online Robust Action Recognition Based on a Hierarchical Model

Temporal Recurrent Networks for Online Action Detection

Spatio-Temporal Attention-Based LSTM Networks for 3D Action Recognition and Detection

Action-Agnostic Human Pose Forecasting

Spatiotemporal Multi-Task Network for Human Activity Understanding.

Online Human Action Recognition with Spatial and Temporal Skeleton Features Using a Distributed Camera Network.

Deep spatiotemporal LSTM network with temporal pattern feature for 3D human action recognition

Multi-Feature Fusion Real-Time Action Recognition Based on 2D to 3D Skeleton

TwinLSTM: Two-channel LSTM Network for Online Action Detection.

Attention-Based Multilevel Co-Occurrence Graph Convolutional LSTM for 3-D Action Recognition

Skeleton-Based Human Action Recognition Using Spatial Temporal 3D Convolutional Neural Networks

Active Temporal Action Detection in Untrimmed Videos Via Deep Reinforcement Learning

Skeleton-based Action Recognition Using LSTM and CNN

Skeleton-based Attention-Aware Spatial-Temporal Model for Action Detection and Recognition.

Efficient Skeleton-Based Action Recognition Via Multi-Stream Depthwise Separable Convolutional Neural Network

Spatial–Temporal Context-Aware Online Action Detection and Prediction

Skeleton Based Temporal Action Detection with YOLO