Abstract:Temporal action proposal (TAP) aims to detect the action instances’ starting and ending times in untrimmed videos, which is fundamental and critical for large-scale video analysis and human action understanding. The main challenge of the temporal action proposal lies in modeling representative temporal relations in long untrimmed videos. Existing state-of-the-art methods achieve temporal modeling by building local-level, proposal-level, or global-level temporal dependencies. Local methods lack a wider receptive field, while proposal and global methods lack the focalization of learning action frames and contain background distractions. In this paper, we propose that learning semantic-level affinities can capture more practical information. Specifically, by modeling semantic associations between frames and action units, action segments (foregrounds) can aggregate supportive cues from other co-occurring actions, and nonaction clips (backgrounds) can learn the discriminations between them and action frames. To this end, we propose a novel framework named the Mask-Guided Network (MGNet) to build semantic-level temporal associations for the TAP task. Specifically, we first propose a Foreground Mask Generation (FMG) module to adaptively generate the foreground mask, representing the locations of the action units throughout the video. Second, we design a Mask-Guided Transformer (MGT) by exploiting the foreground mask to guide the self-attention mechanism to focus on and calculate semantic affinities with the foreground frames. Finally, these two modules are jointly explored in a unified framework. MGNet models the intra-semantic similarities for foregrounds, extracting supportive action cues for boundary refinement; it also builds the inter-semantic distances for backgrounds, providing the semantic gaps to suppress false positives and distractions. Extensive experiments are conducted on two challenging datasets, ActivityNet-1.3 and THUMOS14, and the results demonstrate that our method achieves superior performance.

Search Video Action Proposal with Recurrent and Static YOLO.

YoTube: Searching Action Proposal Via Recurrent and Static Regression Networks

Exploiting Semantic-Level Affinities with a Mask-Guided Network for Temporal Action Proposal in Videos.

A Method of Simultaneously Action Recognition and Video Segmentation of Video Streams.

Searching Action Proposals Via Spatial Actionness Estimation And Temporal Path Inference And Tracking

Unsupervised Action Proposal Ranking through Proposal Recombination

An Active Action Proposal Method Based on Reinforcement Learning

RecapNet: Action Proposal Generation Mimicking Human Cognitive Process

Detecting Temporal Proposal for Action Localization with Tree-structured Search Policy

ProposalVLAD with Proposal-Intra Exploring for Temporal Action Proposal Generation

Play and rewind: Context-aware video temporal action proposals

Content Temporal Relation Network for temporal action proposal generation

Active Temporal Action Detection in Untrimmed Videos Via Deep Reinforcement Learning

SAP: Self-Adaptive Proposal Model for Temporal Action Detection Based on Reinforcement Learning

Temporal Attention Network for Action Proposal

A Proposal-Based Solution to Spatio-Temporal Action Detection in Untrimmed Videos

A Deep Learning Method for Video‐based Action Recognition

Relaxed Transformer Decoders for Direct Action Proposal Generation

CTAP: Complementary Temporal Action Proposal Generation

Spatial–Temporal Context-Aware Online Action Detection and Prediction

Online Action Tube Detection Via Resolving The Spatio-Temporal Context Pattern