Abstract:Depth video sequence-based deep models for recognizing human actions are scarce compared to RGB and skeleton video sequences-based models. This scarcity limits the research advancements based on depth data, as training deep models with small-scale data is challenging. In this work, we propose a sequence classification deep model using depth video data for scenarios when the video data are limited. Unlike summarizing the frame contents of each frame into a single class, our method can directly classify a depth video, i.e., a sequence of depth frames. Firstly, the proposed system transforms an input depth video into three sequences of multi-view temporal motion frames. Together with the three temporal motion sequences, the input depth frame sequence offers a four-stream representation of the input depth action video. Next, the DenseNet121 architecture is employed along with ImageNet pre-trained weights to extract the discriminating frame-level action features of depth and temporal motion frames. The extracted four sets of feature vectors about frames of four streams are fed into four bi-directional (BLSTM) networks. The temporal features are further analyzed through multi-head self-attention (MHSA) to capture multi-view sequence correlations. Finally, the concatenated genre of their outputs is processed through dense layers to classify the input depth video. The experimental results on two small-scale benchmark depth datasets, MSRAction3D and DHA, demonstrate that the proposed framework is efficacious even for insufficient training samples and superior to the existing depth data-based action recognition methods.

A New Multi-channels Sequence Recognition Framework Using Deep Convolutional Neural Network

Motion Sequence Recognition with Multi-sensors Using Deep Convolutional Neural Network.

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Sequence-Based Device-Free Gesture Recognition Framework for Multi-Channel Acoustic Signals

Deep RNN Framework for Visual Sequential Applications

A Spatiotemporal Multi-Channel Learning Framework for Automatic Modulation Recognition

Video Action Recognition Based on Deeper Convolution Networks with Pair-Wise Frame Motion Concatenation

Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network

Feature-Aware Attentive Convolutional Neural Network for Sequence Processing.

A Face Sequence Recognition Method Based on Deep Convolutional Neural Network

A Deep Sequence Learning Framework for Action Recognition in Small-Scale Depth Video Dataset

Underwater Communication Signal Recognition Using Sequence Convolutional Network

An Improved Convolutional Neural Networks Framework with Flexible Channel Shift

A very deep sequences learning approach for human action recognition

A Deep Learning Framework for Signal Detection and Modulation Classification

A Deep Learning Framework for Time Series Classification Using Relative Position Matrix and Convolutional Neural Network.

Video-based Face Recognition Based on Deep Convolutional Neural Network

A Fast Multi-Loss Learning Deep Neural Network for Automatic Modulation Classification

A Spatiotemporal Multi-Stream Learning Framework Based on Attention Mechanism for Automatic Modulation Recognition

Exploiting Multi-Channels Deep Convolutional Neural Networks for Multivariate Time Series Classification

Automatic Modulation Classification Using a Deep Multi-Stream Neural Network