Abstract:Contextual information is essential in action recognition. However, local operations have difficulty in modeling two distant elements, and directly computing the dense relations between any two points brings huge computation and memory burden. Inspired by the recurrent 2D criss-cross attention (RCCA-2D) in image segmentation, we propose a recurrent 3D criss-cross attention (RCCA-3D) that factorizes the global relation map into sparse relation maps to model long-range spatiotemporal context with minor costs for video-based action recognition. Specifically, we first propose a 3D criss-cross attention (CCA-3D) module. Compared with the CCA-2D which only works in space, it can capture the spatiotemporal relationship between the points in the same line along the direction of width, height and time. However, only replacing the two CCA-2Ds in the RCCA-2D with our CCA-3Ds cannot model the spatiotemporal context in videos. Therefore, we further duplicate the CCA-3D with a recurrent mechanism to transmit the relation between the points in a line to a plane and finally to the whole spatiotemporal space. To make the RCCA-3D adaptive for action recognition, we propose a novel recurrent structure rather than directly extending the original 2D structure to 3D. In the experiments, we make a thorough analysis of different structures of RCCA-3D, verifying the proposed structure is more suitable for action recognition. We also compare our RCCA-3D with the non-local attention, showing that the RCCA-3D requires 25% fewer parameters and 30% fewer FLOPs with even higher accuracy. Finally, equipped with our RCCA-3D, 3 networks achieve better and leading performance on 5 RGB-based and skeleton-based datasets.

Higher-order Network for Action Recognition

Learning Visual Context for Group Activity Recognition.

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Learnable Higher-Order Representation for Action Recognition

Revisiting the Spatial and Temporal Modeling for Few-shot Action Recognition

Human Action Recognition with Contextual Constraints Using a RGB-D Sensor

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Actor-Multi-Scale Context Bidirectional Higher Order Interactive Relation Network for Spatial-Temporal Action Localization.

Online Robust Action Recognition Based on a Hierarchical Model

Efficient Spatialtemporal Context Modeling for Action Recognition

Efficient spatiotemporal context modeling for action recognition

Empowering Efficient Spatio-Temporal Learning with a 3D CNN for Pose-Based Action Recognition

Spatial-Temporal Neural Networks For Action Recognition

Learning Heterogeneous Spatial-Temporal Context for Skeleton-Based Action Recognition.

Deep Alternative Neural Network: Exploring Contexts As Early As Possible for Action Recognition

A Novel Hierarchical Framework for Human Action Recognition

Action Recognition By Learning Deep Multi-Granular Spatio-Temporal Video Representation

Spatio-Temporal Adaptive Network with Bidirectional Temporal Difference for Action Recognition

Extracting Hierarchical Spatial and Temporal Features for Human Action Recognition

Temporal Attentive Network for Action Recognition

Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Context