Abstract:Temporal action proposal (TAP) aims to detect the action instances’ starting and ending times in untrimmed videos, which is fundamental and critical for large-scale video analysis and human action understanding. The main challenge of the temporal action proposal lies in modeling representative temporal relations in long untrimmed videos. Existing state-of-the-art methods achieve temporal modeling by building local-level, proposal-level, or global-level temporal dependencies. Local methods lack a wider receptive field, while proposal and global methods lack the focalization of learning action frames and contain background distractions. In this paper, we propose that learning semantic-level affinities can capture more practical information. Specifically, by modeling semantic associations between frames and action units, action segments (foregrounds) can aggregate supportive cues from other co-occurring actions, and nonaction clips (backgrounds) can learn the discriminations between them and action frames. To this end, we propose a novel framework named the Mask-Guided Network (MGNet) to build semantic-level temporal associations for the TAP task. Specifically, we first propose a Foreground Mask Generation (FMG) module to adaptively generate the foreground mask, representing the locations of the action units throughout the video. Second, we design a Mask-Guided Transformer (MGT) by exploiting the foreground mask to guide the self-attention mechanism to focus on and calculate semantic affinities with the foreground frames. Finally, these two modules are jointly explored in a unified framework. MGNet models the intra-semantic similarities for foregrounds, extracting supportive action cues for boundary refinement; it also builds the inter-semantic distances for backgrounds, providing the semantic gaps to suppress false positives and distractions. Extensive experiments are conducted on two challenging datasets, ActivityNet-1.3 and THUMOS14, and the results demonstrate that our method achieves superior performance.

Separately Guided Context-Aware Network for Weakly Supervised Temporal Action Detection

Exploiting Semantic-Level Affinities with a Mask-Guided Network for Temporal Action Proposal in Videos.

ACM-Net: Action Context Modeling Network for Weakly-Supervised Temporal Action Localization

Transferable Knowledge-Based Multi-Granularity Fusion Network for Weakly Supervised Temporal Action Detection

Weakly Supervised Temporal Action Detection with Temporal Dependency Learning

Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization

MCMNET: Multi-Scale Context Modeling Network for Temporal Action Detection

Learning Frame-Level Affinity with Video-Level Labels for Weakly Supervised Temporal Action Detection

Two-Stream Completeness Modeling for Weakly Supervised Temporal Action Detection

Two-Stream Networks for Weakly-Supervised Temporal Action Localization with Semantic-Aware Mechanisms

Superframe-Based Temporal Proposals for Weakly Supervised Temporal Action Detection

Two-stream graph convolutional neural network fusion for weakly supervised temporal action detection

Deep Feature Enhancing and Selecting Network for Weakly Supervised Temporal Action Localization

Weakly-Supervised Temporal Action Localization by Inferring Salient Snippet-Feature

Weakly-Supervised Action Localization by Hierarchical Attention Mechanism with Multi-Scale Fusion Strategies

Spatial–Temporal Context-Aware Online Action Detection and Prediction

A Novel Action Saliency and Context-Aware Network for Weakly-Supervised Temporal Action Localization

Temporal Graph Convolutional Network for Multi-Agent Reinforcement Learning of Action Detection

ContextDet: Temporal Action Detection with Adaptive Context Aggregation

Spatio-Temporal Adaptive Network with Bidirectional Temporal Difference for Action Recognition

Multi-Scale Based Context-Aware Net for Action Detection.