Abstract:Temporal information plays a significant role in video-based human action recognition. How to effectively extract the spatial–temporal characteristics of actions in videos has always been a challenging problem. Most existing methods acquire spatial and temporal cues in videos individually. In this article, we propose a new effective representation for depth video sequences, called hierarchical dynamic depth projected difference images that can aggregate the action spatial and temporal information simultaneously at different temporal scales. We firstly project depth video sequences onto three orthogonal Cartesian views to capture the 3D shape and motion information of human actions. Hierarchical dynamic depth projected difference images are constructed with the rank pooling in each projected view to hierarchically encode the spatial–temporal motion dynamics in depth videos. Convolutional neural networks can automatically learn discriminative features from images and have been extended to video classification because of their superior performance. To verify the effectiveness of hierarchical dynamic depth projected difference images representation, we construct a hierarchical dynamic depth projected difference images–based action recognition framework where hierarchical dynamic depth projected difference images in three views are fed into three identical pretrained convolutional neural networks independently for finely retuning. We design three classification schemes in the framework and different schemes utilize different convolutional neural network layers to compare their effects on action recognition. Three views are combined to describe the actions more comprehensively in each classification scheme. The proposed framework is evaluated on three challenging public human action data sets. Experiments indicate that our method has better performance and can provide discriminative spatial–temporal information for human action recognition in depth videos.

Human action recognition using Adaptive Hierarchical Depth Motion Maps and Gabor filter

Combining Adaptive Hierarchical Depth Motion Maps with Skeletal Joints for Human Action Recognition

Action Recognition Using DMM-Gabor Descriptor and Temporal Segmentation

Recognizing actions using depth motion maps-based histograms of oriented gradients

Human Action Recognition Based on DMMs, HOGs and Contourlet Transform

3D Action Recognition Using Multi-Temporal Depth Motion Maps and Fisher Vector

Combining Multi-scale Directed Depth Motion Maps and Log-Gabor Filters for Human Action Recognition

Multi-Temporal Depth Motion Maps-Based Local Binary Patterns for 3-D Human Action Recognition

CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Action Recognition Using Multi-Scale Histograms of Oriented Gradients Based Depth Motion Trail Images

Action Recognition Based on Depth Motion Map and Hybrid Classifier

Human Action Recognition Using Multilevel Depth Motion Maps

Action Recognition from Depth Sequences Using Weighted Fusion of 2D and 3D Auto-Correlation of Gradients Features

Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences

Action Recognition from Depth Maps Using Deep Convolutional Neural Networks

Multimodal human action recognition based on spatio-temporal action representation recognition model

DMMs-Based Multiple Features Fusion for Human Action Recognition

Online Robust Action Recognition Based on a Hierarchical Model

Hierarchical Dynamic Depth Projected Difference Images–based Action Recognition in Videos with Convolutional Neural Networks

Real-Time Human Action Recognition System Using Depth Map Sequences

Real-Time Human Action Recognition Using DMMs-Based LBP and EOH Features