Abstract:Recent studies of video action recognition can be classified into two categories: the appearance-based methods and the pose-based methods. The appearance-based methods generally cannot model temporal dynamics of large motion well by virtue of optical flow estimation, while the pose-based methods ignore the visual context information such as typical scenes and objects, which are also important cues for action understanding. In this paper, we tackle these problems by proposing a Pose-Appearance Relational Network (PARNet), which models the correlation between human pose and image appearance, and combines the benefits of these two modalities to improve the robustness towards unconstrained real-world videos. There are three network streams in our model, namely pose stream, appearance stream and relation stream. For the pose stream, a Temporal Multi-Pose RNN module is constructed to obtain the dynamic representations through temporal modeling of 2D poses. For the appearance stream, a Spatial Appearance CNN module is employed to extract the global appearance representation of the video sequence. For the relation stream, a Pose-Aware RNN module is built to connect pose and appearance streams by modeling action-sensitive visual context information. Through jointly optimizing the three modules, PARNet achieves superior performances compared with the state-of-the-arts on both the pose-complete datasets (KTH, Penn-Action, UCF11) and the challenging pose-incomplete datasets (UCF101, HMDB51, JHMDB), demonstrating its robustness towards complex environments and noisy skeletons. Its effectiveness on NTU-RGBD dataset is also validated even compared with 3D skeleton-based methods. Furthermore, an appearance-enhanced PARNet equipped with a RGB-based I3D stream is proposed, which outperforms the Kinetics pre-trained competitors on UCF101 and HMDB51. The better experimental results verify the potentials of our framework by integrating various modules.

Temporal Pyramid Relation Network for Video-Based Gesture Recognition

Temporal Pyramid Network for Action Recognition

Spatial and Temporal Pyramid-Based Real-Time Gesture Recognition.

Short-Term Temporal Convolutional Networks for Dynamic Hand Gesture Recognition

Deep learning models beyond temporal frame-wise features for hand gesture video recognition

Long Short-Term Relation Networks for Video Action Detection

High Performance Gesture Recognition Via Effective and Efficient Temporal Modeling.

Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video

Pose-Appearance Relational Modeling for Video Action Recognition

Temporal Reasoning Graph For Activity Recognition

Video Relation Detection with Spatio-Temporal Graph

Temporal-attentive Covariance Pooling Networks for Video Recognition

Temporal Segment Networks for Action Recognition in Videos

A Spatio-Temporal Multilayer Perceptron for Gesture Recognition

Multimodal Spatiotemporal Feature Map for Dynamic Gesture Recognition

Social Relation Recognition from Videos Via Multi-Scale Spatial-Temporal Reasoning

Large-scale Isolated Gesture Recognition Using Pyramidal 3D Convolutional Networks

Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition

Video Visual Relation Detection via 3D Convolutional Neural Network

Video Visual Relation Detection Via 3D Convolutional Neural Network.

Temporal Recursive Propagation Network for Action Recognition