Abstract:Temporal action localization is a challenging task in computer vision, and it tries to find the start time and the end time of the actions and predict their categories. However, compared to temporal action localization, weakly supervised temporal action localization (WTAL) is a more challenging task due to its poor annotations. With only video-level annotation, some background frames, similar to actions, would be classified as actions and produce inaccurate results. In addition, the two-stream fusion problem, ignored previously, also needs to be further considered. To resolve these issues, we propose a novel action saliency and context-aware network (ASCN) for weakly supervised temporal action localization tasks. Specifically, the temporal saliency and context module is designed to enhance the global saliency and context information of the RGB and the flow features to suppress the backgrounds and enhance the actions. In addition, a hybrid attention mechanism using frame differences and two-stream attention is designed to model the local action context information and further enlarge the scores of the potential action regions and suppress the background regions. Finally, to obtain two-stream consistency and solve the fusion problem, we use the similarity loss and a channel self-attention module to adaptively fuse the enhanced RGB and flow features. Extensive experiments demonstrate that ASCN can outperform all of the SOTA WTAL methods on the THUMOS14 dataset and the ActivityNet1.3 dataset with an average mAP that can reach 37.2% on the THUMOS14 dataset and attains an average mAP of 26.3% on the ActivityNet1.3 dataset. On the ActivityNet1.2 dataset, ASCN can also obtain comparable results. Compared with AdapNet (TNNLS20), MMSD (TIP22), and FTCL (CVPR22) on the THUMOS14 dataset, ASCN can outperform them by 13.5%, 2.9%, and 2.8%, respectively.

You watch once more: a more effective CNN architecture for video spatio-temporal action localization

Real-time spatiotemporal action localization algorithm using improved CNNs architecture

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

COWO: Towards Real-Time Spatiotemporal Action Localization in Videos

Revisiting the Spatial and Temporal Modeling for Few-shot Action Recognition

YOWOv2: A Stronger yet Efficient Multi-level Detection Framework for Real-time Spatio-temporal Action Detection

Exploring Stronger Feature for Temporal Action Localization

Spatio-Temporal Fusion Networks for Action Recognition

A Novel Action Saliency and Context-Aware Network for Weakly-Supervised Temporal Action Localization

A fast human action recognition network based on spatio-temporal features

Local-aware spatio-temporal attention network with multi-stage feature fusion for human action recognition

Exploring Frame Segmentation Networks for Temporal Action Localization

YOWO-Plus: An Incremental Improvement

Action-Stage Emphasized Spatiotemporal VLAD for Video Action Recognition

StNet: Local and Global Spatial-Temporal Modeling for Action Recognition

Spatiotemporal Multi-Task Network for Human Activity Understanding.

Action Coherence Network for Weakly-Supervised Temporal Action Localization

Spatial-Temporal Alignment Network for Action Recognition and Detection

Spatio-Temporal Attention Networks for Action Recognition and Detection

Action recognition using attention-based spatio-temporal VLAD networks and adaptive video sequences optimization