Abstract:Temporal action proposal (TAP) aims to detect the action instances’ starting and ending times in untrimmed videos, which is fundamental and critical for large-scale video analysis and human action understanding. The main challenge of the temporal action proposal lies in modeling representative temporal relations in long untrimmed videos. Existing state-of-the-art methods achieve temporal modeling by building local-level, proposal-level, or global-level temporal dependencies. Local methods lack a wider receptive field, while proposal and global methods lack the focalization of learning action frames and contain background distractions. In this paper, we propose that learning semantic-level affinities can capture more practical information. Specifically, by modeling semantic associations between frames and action units, action segments (foregrounds) can aggregate supportive cues from other co-occurring actions, and nonaction clips (backgrounds) can learn the discriminations between them and action frames. To this end, we propose a novel framework named the Mask-Guided Network (MGNet) to build semantic-level temporal associations for the TAP task. Specifically, we first propose a Foreground Mask Generation (FMG) module to adaptively generate the foreground mask, representing the locations of the action units throughout the video. Second, we design a Mask-Guided Transformer (MGT) by exploiting the foreground mask to guide the self-attention mechanism to focus on and calculate semantic affinities with the foreground frames. Finally, these two modules are jointly explored in a unified framework. MGNet models the intra-semantic similarities for foregrounds, extracting supportive action cues for boundary refinement; it also builds the inter-semantic distances for backgrounds, providing the semantic gaps to suppress false positives and distractions. Extensive experiments are conducted on two challenging datasets, ActivityNet-1.3 and THUMOS14, and the results demonstrate that our method achieves superior performance.

Adaptive Proposal Network Based on Generative Adversarial Learning for Weakly Supervised Temporal Sentence Grounding

Weakly Supervised Temporal Sentence Grounding with Gaussian-based Contrastive Proposal Learning

Weakly Supervised Temporal Adjacent Network for Language Grounding

Contrastive Perturbation Network for Weakly Supervised Temporal Sentence Grounding.

Exploiting Semantic-Level Affinities with a Mask-Guided Network for Temporal Action Proposal in Videos.

Efficient Proposal Generation with U-shaped Network for Temporal Sentence Grounding

Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos

Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding.

Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding

TAN: a temporal-aware attention network with context-rich representation for boosting proposal generation

Query-aware Multi-Scale Proposal Network for Weakly Supervised Temporal Sentence Grounding in Videos

Temporal Sentence Grounding in Streaming Videos

Look Closer to Ground Better: Weakly-Supervised Temporal Grounding of Sentence in Video

Video Captioning by Adversarial LSTM

Joint Learning of Local and Global Context for Temporal Action Proposal Generation

To Create What You Tell: Generating Videos from Captions

Collaborative Local-Global Learning for Temporal Action Proposal

Explore Inter-contrast between Videos via Composition for Weakly Supervised Temporal Sentence Grounding.

Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation

Temporal Action Proposal Generation Via Multi-Task Feature Learning.

Fine-Grained Text-to-Video Temporal Grounding from Coarse Boundary