Abstract:Recent studies of video action recognition can be classified into two categories: the appearance-based methods and the pose-based methods. The appearance-based methods generally cannot model temporal dynamics of large motion well by virtue of optical flow estimation, while the pose-based methods ignore the visual context information such as typical scenes and objects, which are also important cues for action understanding. In this paper, we tackle these problems by proposing a Pose-Appearance Relational Network (PARNet), which models the correlation between human pose and image appearance, and combines the benefits of these two modalities to improve the robustness towards unconstrained real-world videos. There are three network streams in our model, namely pose stream, appearance stream and relation stream. For the pose stream, a Temporal Multi-Pose RNN module is constructed to obtain the dynamic representations through temporal modeling of 2D poses. For the appearance stream, a Spatial Appearance CNN module is employed to extract the global appearance representation of the video sequence. For the relation stream, a Pose-Aware RNN module is built to connect pose and appearance streams by modeling action-sensitive visual context information. Through jointly optimizing the three modules, PARNet achieves superior performances compared with the state-of-the-arts on both the pose-complete datasets (KTH, Penn-Action, UCF11) and the challenging pose-incomplete datasets (UCF101, HMDB51, JHMDB), demonstrating its robustness towards complex environments and noisy skeletons. Its effectiveness on NTU-RGBD dataset is also validated even compared with 3D skeleton-based methods. Furthermore, an appearance-enhanced PARNet equipped with a RGB-based I3D stream is proposed, which outperforms the Kinetics pre-trained competitors on UCF101 and HMDB51. The better experimental results verify the potentials of our framework by integrating various modules.

Pose-Promote: Progressive Visual Perception for Activities of Daily Living

Pose-aware video action segmentation

PVO: Panoptic Visual Odometry.

Follow the Attention: Combining Partial Pose and Object Motion for Fine-Grained Action Detection

MOTPose: Multi-object 6D Pose Estimation for Dynamic Video Sequences using Attention-based Temporal Fusion

YOLOv8-PoseBoost: Advancements in Multimodal Robot Pose Keypoint Detection

Pose for Action - Action for Pose

UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing

VividPose: Advancing Stable Video Diffusion for Realistic Human Image Animation

VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Living

PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling

Pose-conditioned Spatio-Temporal Attention for Human Action Recognition

Pose-Appearance Relational Modeling for Video Action Recognition

Joint Action Recognition And Pose Estimation From Video

A Pose Estimation Algorithm for Multimodal Data Fusion

Kpose: A New Representation For Action Recognition

An Approach to Pose-Based Action Recognition

Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformers

Exploring discriminative pose sub-patterns for effective action classification.

EPAM-Net: An Efficient Pose-driven Attention-guided Multimodal Network for Video Action Recognition

Skeleton Focused Human Activity Recognition in RGB Video