Abstract:Human activity understanding with 3D/depth sensors has received increasing attention in multimedia processing and interactions. This work targets on developing a novel deep model for automatic activity recognition from RGB-D videos. We represent each human activity as an ensemble of cubic-like video segments, and learn to discover the temporal structures for a category of activities, i.e. how the activities to be decomposed in terms of classification. Our model can be regarded as a structured deep architecture, as it extends the convolutional neural networks (CNNs) by incorporating structure alternatives. Specifically, we build the network consisting of 3D convolutions and max-pooling operators over the video segments, and introduce the latent variables in each convolutional layer manipulating the activation of neurons. Our model thus advances existing approaches in two aspects: (i) it acts directly on the raw inputs (grayscale-depth data) to conduct recognition instead of relying on hand-crafted features, and (ii) the model structure can be dynamically adjusted accounting for the temporal variations of human activities, i.e. the network configuration is allowed to be partially activated during inference. For model training, we propose an EM-type optimization method that iteratively (i) discovers the latent structure by determining the decomposed actions for each training example, and (ii) learns the network parameters by using the back-propagation algorithm. Our approach is validated in challenging scenarios, and outperforms state-of-the-art methods. A large human activity database of RGB-D videos is presented in addition.

Human Action Recognition Using a Distributed RGB-Depth Camera Network

3D Point-to-Keypoint Voting Network for 6D Pose Estimation

Human Action Recognition with Contextual Constraints Using a RGB-D Sensor

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences

Online Robust Action Recognition Based on a Hierarchical Model

RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks

DC3D: A Video Action Recognition Network Based on Dense Connection

Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition

Deep Attention Network for Egocentric Action Recognition.

3DFCNN: Real-Time Action Recognition using 3D Deep Neural Networks with Raw Depth Information

Empowering Efficient Spatio-Temporal Learning with a 3D CNN for Pose-Based Action Recognition

Joint Human Detection and Head Pose Estimation Via Multistream Networks for RGB-D Videos

Residual Frames with Efficient Pseudo-3D CNN for Human Action Recognition

Human Motion Tracking by Multiple RGBD Cameras.

Action Recognition In Rgb-D Egocentric Videos

Typing Video frames after person detection Pose Tube 2 D Deconv Score fusion RGB action recognition Pose action recognition Pose estimation

Real-Time Human Action Recognition System Using Depth Map Sequences

Action Recognition from Depth Sequences Using Weighted Fusion of 2D and 3D Auto-Correlation of Gradients Features

3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks