Abstract:Abstract Fusion of multiple modalities from different sensors is an important area of research for multimodal human action recognition. In this paper, we conduct an in-depth study to investigate the effect of different parameters like input preprocessing, data augmentation, network architectures and model fusion so as to come up with a practical guideline for multimodal action recognition using deep learning paradigm. First, for RGB videos, we propose a novel image-based descriptor called stacked dense flow difference image (SDFDI), capable of capturing the spatio-temporal information present in a video sequence. A variety of deep 2D convolutional neural networks (CNN) are then trained to compare our SDFDI against state-of-the-art image-based representations. Second, for skeleton stream, we propose data augmentation technique based on 3D transformations so as to facilitate training a deep neural network on small datasets. We also propose a bidirectional gated recurrent unit (BiGRU) based recurrent neural network (RNN) to model skeleton data. Third, for inertial sensor data, we propose data augmentation based on jittering with white Gaussian noise along with deep a 1D-CNN network for action classification. The outputs of all these three heterogeneous networks (1D-CNN, 2D-CNN and BiGRU) are combined by a variety of model fusion approach based on score and feature fusion. Finally, in order to illustrate the efficacy of the proposed framework, we test our model on a publicly available UTD-MHAD dataset, and achieved an overall accuracy of 97.91%, which is about 4% higher than using each modality individually. We hope that the discussions and conclusions from this work will provide a deeper insight to the researchers in the related fields, and provide avenues for further studies for different multi-sensor based fusion architectures.

View-invariant Deep Architecture for Human Action Recognition using late fusion

View-invariant Human Action Recognition Via Robust Locally Adaptive Multi-View Learning

View-invariant feature discovering for multi-camera human action recognition

View-independent Representation with Frame Interpolation Method for Skeleton-Based Human Action Recognition

B2C-AFM: Bi-Directional Co-Temporal and Cross-Spatial Attention Fusion Model for Human Action Recognition.

Discriminative Multi-View Dynamic Image Fusion for Cross-View 3-D Action Recognition

Multi-View Region Adaptive Multi-temporal DMM and RGB Action Recognition

Evaluating fusion of RGB-D and inertial sensors for multimodal human action recognition

Fusing Shape and Motion Matrices for View Invariant Action Recognition Using 3D Skeletons

A deep multimodal network based on bottleneck layer features fusion for action recognition

Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures

Part-wise Spatio-temporal Attention Driven CNN-based 3D Human Action Recognition

Human action recognition using late fusion and dimensionality reduction

Two Stream LSTM: A Deep Fusion Framework for Human Action Recognition

Real-time Action Recognition by Feature-Level Fusion of Depth and Inertial Sensor

Deeply Learned View-Invariant Features for Cross-View Action Recognition

Deep learning-based multi-view 3D-human action recognition using skeleton and depth data

Multi-view key information representation and multi-modal fusion for single-subject routine action recognition

A Novel Two Stream Decision Level Fusion of Vision and Inertial Sensors Data for Automatic Multimodal Human Activity Recognition System

Spatio-Temporal Attention Deep Network for Skeleton Based View-Invariant Human Action Recognition

Multiple Stream Deep Learning Model for Human Action Recognition