Abstract:Purpose The purpose of this paper is to provide a fast and accurate network for spatiotemporal action localization in videos. It detects human actions both in time and space simultaneously in real-time, which is applicable in real-world scenarios such as safety monitoring and collaborative assembly. Design/methodology/approach This paper design an end-to-end deep learning network called collaborator only watch once (COWO). COWO recognizes the ongoing human activities in real-time with enhanced accuracy. COWO inherits from the architecture of you only watch once (YOWO), known to be the best performing network for online action localization to date, but with three major structural modifications: COWO enhances the intraclass compactness and enlarges the interclass separability in the feature level. A new correlation channel fusion and attention mechanism are designed based on the Pearson correlation coefficient. Accordingly, a correction loss function is designed. This function minimizes the same class distance and enhances the intraclass compactness. Use a probabilistic K-means clustering technique for selecting the initial seed points. The idea behind this is that the initial distance between cluster centers should be as considerable as possible. CIOU regression loss function is applied instead of the Smooth L1 loss function to help the model converge stably. Findings COWO outperforms the original YOWO with improvements of frame mAP 3% and 2.1% at a speed of 35.12 fps. Compared with the two-stream, T-CNN, C3D, the improvement is about 5% and 14.5% when applied to J-HMDB-21, UCF101-24 and AGOT data sets. Originality/value COWO extends more flexibility for assembly scenarios as it perceives spatiotemporal human actions in real-time. It contributes to many real-world scenarios such as safety monitoring and collaborative assembly.

COWO: Towards Real-Time Spatiotemporal Action Localization in Videos

Real-time spatiotemporal action localization algorithm using improved CNNs architecture

You watch once more: a more effective CNN architecture for video spatio-temporal action localization

YOWOv2: A Stronger yet Efficient Multi-level Detection Framework for Real-time Spatio-temporal Action Detection

Spatiotemporal Multi-Task Network for Human Activity Understanding.

You Ought to Look Around: Precise, Large Span Action Detection.

Spatiotemporal Action Recognition in Restaurant Videos

Spatial–Temporal Context-Aware Online Action Detection and Prediction

A fast human action recognition network based on spatio-temporal features

Precise Temporal Action Localization by Evolving Temporal Proposals

Local-aware spatio-temporal attention network with multi-stage feature fusion for human action recognition

Multi‐scale feature learning and temporal probing strategy for one‐stage temporal action localization

Online Human Action Recognition with Spatial and Temporal Skeleton Features Using a Distributed Camera Network.

Spatio-Temporal Action Localization in a Weakly Supervised Setting

Action Recognition and Localization with Instance FCNN

Cascading Spatio-Temporal Attention Network for Real-Time Action Detection

Temporal Action Localization in the Deep Learning Era: A Survey.

Spatiotemporal Saliency Representation Learning for Video Action Recognition

Collaborative Spatio-temporal Feature Learning for Video Action Recognition

A weakly supervised CNN model for spatial localization of human activities in unconstraint environment

YOWO-Plus: An Incremental Improvement