Abstract:Existing action detection algorithms usually generate action proposals through an extensive search over the video at multiple temporal scales, which brings about huge computational overhead and deviates from the human perception procedure. We argue that the process of detecting actions in the video should be naturally one of observation and refinement: observe the current temporal window and refine the span of attended window to cover true action regions. In this paper, we propose an active action detection model that learns to search actions through continuously adjusting the bounds of temporal attended window in a self-adaptive way. The whole process can be deemed as an exploring procedure, where an agent is first placed at the beginning of the video and then traverses the whole video by adopting a sequence of transformations on the current attended window to discover actions according to a learned policy. We utilize reinforcement learning, especially the deep Q-learning algorithm to learn the agent's decision policy. Actually, we construct an end-to-end trainable framework for the action detection task, which includes a proposal generation network based on deep Q-learning, and the classification and regression networks responsible for the action category prediction and the action location adjustment, respectively. In addition, we design a long short-term memory structure upon extracted convolutional neural network features of sparsely sampled frames to generate the effective feature representations for video sequences of various durations. We evaluate the action proposal performance of our approach on THUMOS'14 and assess the generalization ability for unseen action categories on ActivityNet. We also compare the action detection performance of ours with other state-of-the-art methods on both data sets. Experiment results validate the effectiveness of the proposed approach, which can achieve comparative or superior performance than other action detection methods via much fewer proposals.

Improving accuracy of temporal action detection by deep hybrid convolutional network

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

DCAN: Improving Temporal Action Detection Via Dual Context Aggregation

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Cascaded Boundary Network for High-Quality Temporal Action Proposal Generation

Exploring Hybrid Spatio-Temporal Convolutional Networks for Human Action Recognition.

A Spatio-temporal Hybrid Network for Action Recognition

Improving Temporal Action Proposal Generation by Using High Performance Computing.

A Temporal Action Detection Model Based on Deep Reinforcement Learning.

Proposal Complementary Action Detection

A New Temporal Deconvolutional Pyramid Network For Action Detection

Graph Attention Based Proposal 3D ConvNets for Action Detection.

Active Temporal Action Detection in Untrimmed Videos Via Deep Reinforcement Learning

Temporal action proposal for online driver action monitoring using Dilated Convolutional Temporal Prediction Network

Temporal Action Detection Based on Temporal Convolutional Network with Coarse-Grained Clips Construction

Faster-TAD: Towards Temporal Action Detection with Proposal Generation and Classification in a Unified Network

Multi-level Temporal Pyramid Network for Action Detection

Bi-direction Feature Pyramid Temporal Action Detection Network

STH: Spatio-Temporal Hybrid Convolution for Efficient Action Recognition

Hybrid Attentive Prototypical Network for Few-Shot Action Recognition

An Attention-Based Bidirectional GRU Network for Temporal Action Proposals Generation