Abstract:In this paper, we present a method (Action-Fusion) for human action recognition from depth maps and posture data using convolutional neural networks (CNNs). Two input descriptors are used for action representation. The first input is a depth motion image that accumulates consecutive depth maps of a human action, whilst the second input is a proposed moving joints descriptor which represents the motion of body joints over time. In order to maximize feature extraction for accurate action classification, three CNN channels are trained with different inputs. The first channel is trained with depth motion images (DMIs), the second channel is trained with both DMIs and moving joint descriptors together, and the third channel is trained with moving joint descriptors only. The action predictions generated from the three CNN channels are fused together for the final action classification. We propose several fusion score operations to maximize the score of the right action. The experiments show that the results of fusing the output of three channels are better than using one channel or fusing two channels only. Our proposed method was evaluated on three public datasets: 1) Microsoft action 3-D dataset (MSRAction3D); 2) University of Texas at Dallas-multimodal human action dataset; and 3) multimodal action dataset (MAD) dataset. The testing results indicate that the proposed approach outperforms most of existing state-of-the-art methods, such as histogram of oriented 4-D normals and Actionlet on MSRAction3D. Although MAD dataset contains a high number of actions (35 actions) compared to existing action RGB-D datasets, this paper surpasses a state-of-the-art method on the dataset by 6.84%.

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Action Recognition with Joints-Pooled 3D Deep Convolutional Descriptors

Action Recognition Algorithm Based on 2D Human Pose Estimation Method

Human Action Recognition Using Deep Learning Methods.

Human Pose Estimation Based on Step Deep Convolution Neural Network.

Body Joint Guided 3-D Deep Convolutional Descriptors for Action Recognition

Human Action Recognition From Digital Videos Based on Deep Learning.

Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures

Recognizing Human Actions As the Evolution of Pose Estimation Maps

Exploiting Pose Mask Features For Video Action Recognition

Learning Human Pose Models from Synthesized Data for Robust RGB-D Action Recognition

Human Body Gesture Recognition Method Based on Deep Learning

A Multi-Task Neural Network for Action Recognition with 3D Key-Points.

Human Pose Estimation Based on Feature Enhancement and Multi-Scale Feature Fusion

Skeleton-Indexed Deep Multi-Modal Feature Learning for High Performance Human Action Recognition

An Approach to Pose-Based Action Recognition

Recognizing actions in images by fusing multiple body structure cues

Empowering Efficient Spatio-Temporal Learning with a 3D CNN for Pose-Based Action Recognition

Image-based Action Recognition Using Hint-Enhanced Deep Neural Networks

Pose-Guided Inflated 3D ConvNet for Action Recognition in Videos

Human Action Recognition by Learning Spatio-Temporal Features with Deep Neural Networks