Abstract:Weakly supervised temporal action localization (WSTAL) aims to localize actions in untrimmed videos using video-level labels. Despite recent advances, existing approaches mainly follow a localization-by-classification pipeline, generally processing each segment individually, thereby exploiting only limited contextual information. As a result, the model will lack a comprehensive understanding (e.g. appearance and temporal structure) of various action patterns, leading to ambiguity in classification learning and temporal localization. Our work addresses this from a novel perspective, by exploring and exploiting the cross-video contextual knowledge within the dataset to recover the dataset-level semantic structure of action instances via weak labels only, thereby indirectly improving the holistic understanding of fine-grained action patterns and alleviating the aforementioned ambiguities. Specifically, an end-to-end framework is proposed, including a Robust Memory-Guided Contrastive Learning (RMGCL) module and a Global Knowledge Summarization and Aggregation (GKSA) module. First, the RMGCL module explores the contrast and consistency of cross-video action features, assisting in learning more structured and compact embedding space, thus reducing ambiguity in classification learning. Further, the GKSA module is used to efficiently summarize and propagate the cross-video representative action knowledge in a learnable manner to promote holistic action patterns understanding, which in turn allows the generation of high-confidence pseudo-labels for self-learning, thus alleviating ambiguity in temporal localization. Extensive experiments on THUMOS14, ActivityNet1.3, and FineAction demonstrate that our method outperforms the state-of-the-art methods, and can be easily plugged into other WSTAL methods.

Vectorized Evidential Learning for Weakly-supervised Temporal Action Localization

Entropy Guided Attention Network for Weakly-Supervised Action Localization.

Probabilistic Vision-Language Representation for Weakly Supervised Temporal Action Localization

Weakly Supervised Temporal Action Localization Through Contrastive Learning

Generalized Uncertainty-Based Evidential Fusion with Hybrid Multi-Head Attention for Weak-Supervised Temporal Action Localization

Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning

Weakly Supervised Temporal Action Localization through Contrast based Evaluation Networks

Weakly supervised temporal action localization: a survey

DeTAL: Open-Vocabulary Temporal Action Localization with Decoupled Networks

Action Sensitivity Learning for Temporal Action Localization

Cross-Video Contextual Knowledge Exploration and Exploitation for Ambiguity Reduction in Weakly Supervised Temporal Action Localization

Active learning with effective scoring functions for semi-supervised temporal action localization

Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization

Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization

Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks?

Video-Specific Query-Key Attention Modeling for Weakly-Supervised Temporal Action Localization

Learning Temporal Co-Attention Models for Unsupervised Video Action Localization.

Adaptive Prototype Learning for Weakly-supervised Temporal Action Localization

Weakly-Supervised Temporal Action Localization Through Local-Global Background Modeling

Learnable Feature Augmentation Framework for Temporal Action Localization

VAL: Visual-Attention Action Localizer