Abstract:Abstract Fusion of multiple modalities from different sensors is an important area of research for multimodal human action recognition. In this paper, we conduct an in-depth study to investigate the effect of different parameters like input preprocessing, data augmentation, network architectures and model fusion so as to come up with a practical guideline for multimodal action recognition using deep learning paradigm. First, for RGB videos, we propose a novel image-based descriptor called stacked dense flow difference image (SDFDI), capable of capturing the spatio-temporal information present in a video sequence. A variety of deep 2D convolutional neural networks (CNN) are then trained to compare our SDFDI against state-of-the-art image-based representations. Second, for skeleton stream, we propose data augmentation technique based on 3D transformations so as to facilitate training a deep neural network on small datasets. We also propose a bidirectional gated recurrent unit (BiGRU) based recurrent neural network (RNN) to model skeleton data. Third, for inertial sensor data, we propose data augmentation based on jittering with white Gaussian noise along with deep a 1D-CNN network for action classification. The outputs of all these three heterogeneous networks (1D-CNN, 2D-CNN and BiGRU) are combined by a variety of model fusion approach based on score and feature fusion. Finally, in order to illustrate the efficacy of the proposed framework, we test our model on a publicly available UTD-MHAD dataset, and achieved an overall accuracy of 97.91%, which is about 4% higher than using each modality individually. We hope that the discussions and conclusions from this work will provide a deeper insight to the researchers in the related fields, and provide avenues for further studies for different multi-sensor based fusion architectures.

Human-Object Contour for Action Recognition with Attentional Multi-modal Fusion Network

B2C-AFM: Bi-Directional Co-Temporal and Cross-Spatial Attention Fusion Model for Human Action Recognition.

Local-aware spatio-temporal attention network with multi-stage feature fusion for human action recognition

Human-centric multimodal fusion network for robust action recognition

Human Action Recognition Based on Improved Fusion Attention CNN and RNN

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition

Human Action Recognition Based on Hierarchical Multi-Scale Adaptive Conv-Long Short-Term Memory Network

Multi-modality Fusion Network for Action Recognition.

Using a Selective Ensemble Support Vector Machine to Fuse Multimodal Features for Human Action Recognition

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Towards Improved Human Action Recognition Using Convolutional Neural Networks and Multimodal Fusion of Depth and Inertial Sensor Data

Multimodal human action recognition based on spatio-temporal action representation recognition model

Human Interaction Recognition Method Based on Parallel Multi-Feature Fusion Network

Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures

Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial Sensors

Evaluating fusion of RGB-D and inertial sensors for multimodal human action recognition

DMMs-Based Multiple Features Fusion for Human Action Recognition

Human Action Recognition Based on HOIRM Feature Fusion and AP Clustering BOW.

Realistic Human Action Recognition With Multimodal Feature Selection and Fusion

A fast human action recognition network based on spatio-temporal features