Abstract:Action prediction is an important task in human activity analysis, which has many practical applications, such as human–robot interactions and autonomous driving. Action prediction often comprises two subtasks: action semantic prediction and future human motion prediction. Most of the existing works treat these subtasks separately, ignoring the correlations, leading to unsatisfying performance. By contrast, we jointly model these tasks and improve human motion predictions utilizing their action semantics. In terms of methodology, we propose a novel multi-task framework (Multi-head TrajectoryCNN) to simultaneously predict the action semantics and human motion of future human movements. Specifically, we first extract a general spatiotemporal representation of partial observations via two regression blocks. Then, we propose a regression head and a classification head for predicting future human motion and action semantics of human motion, respectively. For the regression head, another two stacked regression blocks and two convolutional layers are applied to predict future poses from the general representation learning. For the classification head, we propose a classification block and stack two regression blocks to predict action semantics from the general representation. In this way, the regression and classification heads are incorporated into a unified framework. During the backward propagation of the network, the human motion prediction and the semantic prediction may be enhanced by each other. NTU RGB+D is a widely used large-scale dataset for action recognition, which was collected by 40 different subjects from three views. Based on the official protocols, we use the skeletal modality and process action sequences with fixed lengths for the evaluation of our action prediction task. Experiments on NTU RGB+D show our model’s state-of-the-art performance. Furthermore, the experimental results also show that semantic information is of great help in predicting future human motion.

A Multi-Task Neural Network for Action Recognition with 3D Key-Points.

3D Point-to-Keypoint Voting Network for 6D Pose Estimation

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

KAN-HyperpointNet for Point Cloud Sequence-Based 3D Human Action Recognition

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Pose-Guided Inflated 3D ConvNet for Action Recognition in Videos

Dual-view 3D Human Pose Estimation Without Camera Parameters for Action Recognition

Multi-Person Pose Estimation in the Wild: Using Adversarial Method to Train a Top-Down Pose Estimation Network

A Fine-to-Coarse Convolutional Neural Network for 3D Human Action Recognition

Multi-Feature Fusion Real-Time Action Recognition Based on 2D to 3D Skeleton

Human Action Recognition Based on Kinematic Similarity in Real Time.

Empowering Efficient Spatio-Temporal Learning with a 3D CNN for Pose-Based Action Recognition

Center point to pose: Multiple views 3D human pose estimation for multi-person

Multi-task neural network with physical constraint for real-time multi-person 3D pose estimation from monocular camera

Multi-Head TrajectoryCNN: A New Multi-Task Framework for Action Prediction

Dual networks based 3D Multi-Person Pose Estimation from Monocular Video

Keypoints-based Multimodal Network for Robust Human Action Recognition

Action recognition method based on a novel keyframe extraction method and enhanced 3D convolutional neural network

Channel attention and multi-scale graph neural networks for skeleton-based action recognition

Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences

End-to-end Learning of Deep Convolutional Neural Network for 3D Human Action Recognition