Abstract:In this paper, we present a method (Action-Fusion) for human action recognition from depth maps and posture data using convolutional neural networks (CNNs). Two input descriptors are used for action representation. The first input is a depth motion image that accumulates consecutive depth maps of a human action, whilst the second input is a proposed moving joints descriptor which represents the motion of body joints over time. In order to maximize feature extraction for accurate action classification, three CNN channels are trained with different inputs. The first channel is trained with depth motion images (DMIs), the second channel is trained with both DMIs and moving joint descriptors together, and the third channel is trained with moving joint descriptors only. The action predictions generated from the three CNN channels are fused together for the final action classification. We propose several fusion score operations to maximize the score of the right action. The experiments show that the results of fusing the output of three channels are better than using one channel or fusing two channels only. Our proposed method was evaluated on three public datasets: 1) Microsoft action 3-D dataset (MSRAction3D); 2) University of Texas at Dallas-multimodal human action dataset; and 3) multimodal action dataset (MAD) dataset. The testing results indicate that the proposed approach outperforms most of existing state-of-the-art methods, such as histogram of oriented 4-D normals and Actionlet on MSRAction3D. Although MAD dataset contains a high number of actions (35 actions) compared to existing action RGB-D datasets, this paper surpasses a state-of-the-art method on the dataset by 6.84%.

DMM-Pyramid Based Deep Architectures for Action Recognition with Depth Cameras

Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences

3D Convolutional Neural Network for Action Recognition.

Deep learning-based multi-view 3D-human action recognition using skeleton and depth data

Action Recognition for Depth Video using Multi-view Dynamic Images

3DFCNN: Real-Time Action Recognition using 3D Deep Neural Networks with Raw Depth Information

Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures

Human Action Recognition Using Deep Learning Methods.

Human Action Recognition From Digital Videos Based on Deep Learning.

CNN-based and DTW features for human activity recognition on depth maps

Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks

Skeleton-Based Human Action Recognition Using Spatial Temporal 3D Convolutional Neural Networks

RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks

Action Recognition from Depth Sequences Using Weighted Fusion of 2D and 3D Auto-Correlation of Gradients Features

D3D: Dual 3-D Convolutional Network for Real-Time Action Recognition

Skeleton-based Action Recognition Using LSTM and CNN

Continuous Motion Recognition In Depth Camera Based On Recurrent Neural Networks And Grid-Based Average Depth

Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition

End-to-end Learning of Deep Convolutional Neural Network for 3D Human Action Recognition

Online Robust Action Recognition Based on a Hierarchical Model

A Fine-to-Coarse Convolutional Neural Network for 3D Human Action Recognition