Abstract:Due to the sparse single-frame annotations, current Single-Frame Temporal Action Localization (SF-TAL) methods generally employ threshold-based pseudo-label generation strategies. However, these approaches suffer from inefficient data utilization, as only parts of unlabeled frames with confidence scores surpassing a predefined threshold are selected for training. Moreover, the variability of single-frame annotations and unreliable model predictions introduce pseudo-label noise. To address these challenges, we propose two strategies by using the relationship of the video segments with their neighbors': 1) temporal neighbor-guided soft pseudo-label generation (TNPG); and 2) semantic neighbor-guided pseudo-label refinement (SNPR). TNPG utilizes a local-global self-attention mechanism in a transformer encoder to capture temporal neighbor information while focusing on the whole video. Then the generated self-attention map is multiplied by the network predictions to propagate information between labeled and unlabeled frames, and produce soft pseudo-label for all segments. Despite this, label noise persists due to unreliable model predictions. To mitigate this, SNPR refines pseudo-labels based on the assumption that predictions should resemble their semantic nearest neighbors'. Specifically, we search for semantic nearest neighbors of each video segment by cosine similarity in the feature space. Then the refined soft pseudo-labels can be obtained by a weight combination of the original pseudo-label and the semantic nearest neighbors'. Finally, the model can be trained with the refined pseudo-labels, and the performance has been greatly improved. Comprehensive experimental results on different benchmarks show that we achieve state-of-the-art performances on THUMOS14, ActivityNet1.2, and ActivityNet1.3 datasets.

Temporal Label Aggregation for Unintentional Action Localization

Probabilistic Temporal Modeling for Unintentional Action Localization

Entropy Guided Attention Network for Weakly-Supervised Action Localization.

Uncertainty Guided Collaborative Training for Weakly Supervised Temporal Action Detection

Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos

Revisiting Unsupervised Temporal Action Localization: the Primacy of High-Quality Actionness and Pseudolabels

Modeling Sub-Actions for Weakly Supervised Temporal Action Localization

ADM-Loc: Actionness Distribution Modeling for Point-supervised Temporal Action Localization

Learning Reliable Dense Pseudo-Labels for Point-Level Weakly-Supervised Action Localization

Unsupervised Temporal Action Localization Via Self-paced Incremental Learning

Towards Adaptive Pseudo-label Learning for Semi-Supervised Temporal Action Localization

Learning from Noisy Pseudo Labels for Semi-Supervised Temporal Action Localization

Weakly-Supervised Temporal Action Localization Based on Attention Regularization

Uncertainty-Based Spatial-Temporal Attention for Online Action Detection.

Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions

Learning Temporal Co-Attention Models for Unsupervised Video Action Localization.

Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modeling

Multilevel Semantic and Adaptive Actionness Learning for Weakly Supervised Temporal Action Localization

STAT: Towards Generalizable Temporal Action Localization

Adaptive Mutual Supervision for Weakly-Supervised Temporal Action Localization

Neighbor-Guided Pseudo-Label Generation and Refinement for Single-Frame Supervised Temporal Action Localization