Abstract:Human Activity Recognition(HAR) plays an important role in the field of ubiquitous computing, which can benefit various human-centric applications such as smart homes, health monitoring, and aging systems. Human Activity Recognition mainly leverages smartphones and wearable devices to collect sensory signals labeled with activity annotations and train machine learning models to recognize individuals' activity automatically. In order to deploy the Human Activity Recognition model in real-world scenarios, however, there are two major barriers. Firstly, sensor data and activity labels are traditionally collected using special experimental equipment in a controlled environment, which means fitting models trained with these datasets may result in poor generalization to real-life scenarios. Secondly, existing studies focus on single or a few modalities of sensor readings, which neglect useful information and its relations existing in multimodal sensor data. To tackle these issues, we propose a novel activity recognition model for multimodal sensory data fusion: Marfusion, and an experimental data collection platform for HAR tasks in real-world scenarios: MarSense. Specifically, Marfusion extensively uses a convolution structure to extract sensory features for each modality of the smartphone sensor and then fuse the multimodal features using the attention mechanism. MarSense can automatically collect a large amount of smartphone sensor data via smartphones among multiple users in their natural-used conditions and environment. To evaluate our proposed platform and model, we conduct a data collection experiment in real-life among university students and then compare our Marfusion model with several other state-of-the-art models on the collected datasets. Experimental Results do not only indicate that the proposed platform collected Human Activity Recognition data in the real-world scenario successfully, but also verify the advantages of the Marfusion model compared to existing models in Human Activity Recognition.

A Multiscale Cross-Modal Interactive Fusion Network for Human Activity Recognition Using Wearable Sensors and Smartphones

Wearable Sensor Based Multimodal Human Activity Recognition Exploiting the Diversity of Classifier Ensemble.

Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity Recognition

Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things

Privacy-preserving Activity Recognition Using Multimodal Sensors in Smart Office

Human Activity Recognition Based on Multi-Modal Fusion

A Multi-dimensional Parallel Convolutional Connected Network Based on Multi-source and Multi-modal Sensor Data for Human Activity Recognition

A Multidimensional Parallel Convolutional Connected Network Based on Multisource and Multimodal Sensor Data for Human Activity Recognition

CIR-DFENet: Incorporating Cross-Modal Image Representation and Dual-Stream Feature Enhanced Network for Activity Recognition

Marfusion: An Attention-Based Multimodal Fusion Model for Human Activity Recognition in Real-World Scenarios

Human activity recognition through deep learning: Leveraging unique and common feature fusion in wearable multi-sensor systems

Outer Product-Based Fusion of Smartwatch Sensor Data for Human Activity Recognition

Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Multiscale Deep Feature Learning for Human Activity Recognition Using Wearable Sensors

A multi-scale feature extraction fusion model for human activity recognition

Multiresolution Fusion Convolutional Network for Open Set Human Activity Recognition

Multi-resolution Fusion Convolutional Network for Open Set Human Activity Recognition

Multi-STMT: Multi-Level Network for Human Activity Recognition Based on Wearable Sensors

Multi-scale Context-Aware Networks Based on Fragment Association for Human Activity Recognition

DiamondNet: A Neural-Network-Based Heterogeneous Sensor Attentive Fusion for Human Activity Recognition

Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition