Abstract:Convolutional Neural Networks (CNNs) usually use top-level appearance features of video frames for action recognition. However, these methods discard the implicit complementary advantages across different-scale appearance representations which are effective for object detection, instance segmentation and person re-identification. In this paper, a new spatial pyramid module is proposed to take full use of inherent multi-scale information of CNNs with nearly cost-free by which a bottom-up architecture with lateral connections is constructed for combining high-, mid-, low-level representations of CNNs into a hierarchical frame-level feature elaborately. Additionally, temporal relations at appropriate timescale are contributed to the identification of an action. To this end, we also propose a new temporal pyramid module in which frame-level features belonged to one video are reused by various timescale pooling approaches to get different time-grained features of snippets efficiently. Followed by snippet-relation reasoning, different timescale temporal relations are derived and accumulated for the comprehensive prediction. Unifying the proposed spatial and temporal pyramid modules, a novel network, Spatial-Temporal Pyramid Network (S-TPNet), is proposed to extract spatial-temporal pyramid features for action recognition in videos. Unlike previous models which boost performance at the cost of computation, S-TPNet can be trained in an end-to-end fashion with great efficiency. Extensive experiments on Kinetics, UCF101, and HMDB51 demonstrate that S-TPNet displays significant performance improvements compared with existing frameworks and obtains competitive performance with the state-of-the-arts.(1) (C) 2019 Elsevier B.V. All rights reserved.

Action Recognition Based on Spatial-Temporal Pyramid Sparse Coding.

Spatio-temporal Laplacian Pyramid Coding for Action Recognition.

Recognizing Actions Via Sparse Coding on Structure Projection

Local Spatiotemporal Coding and Sparse Representation Based Human Action Recognition

Action Recognition by Spatio-Temporal Oriented Energies

Sparse Coding on Local Spatial-Temporal Volumes for Human Action Recognition

Action Recognition Using Local Consistent Group Sparse Coding with Spatio-Temporal Structure.

Beyond Spatial Pyramid Matching: Space-time Extended Descriptor for Action Recognition

Action recognition method based on sparse coding local spatio-temporal descriptors

Modeling Geometric-Temporal Context with Directional Pyramid Co-Occurrence for Action Recognition

Spatio-temporal cuboid pyramid for action recognition using depth motion sequences

Sparse Coding-Based Spatiotemporal Saliency for Action Recognition

Action Recognition Based on Spatio-temporal Log-Euclidean Covariance Matrix

Action Recognition Via Structured Codebook Construction

Spatio-Temporal Collaborative Module for Efficient Action Recognition

Spatial-temporal Pyramid Based Convolutional Neural Network for Action Recognition

Hierarchical and Spatio-Temporal Sparse Representation for Human Action Recognition.

Sparse Representations of Clustered Video Shots for Action Recognition

Spatial-Temporal Pyramid Graph Reasoning for Action Recognition

Human action recognition using pyramid vocabulary tree

Action Recognition with Spatial-Temporal Representation Analysis Across Grassmannian Manifold and Euclidean Space