Abstract:Human Activity Recognition(HAR) plays an important role in the field of ubiquitous computing, which can benefit various human-centric applications such as smart homes, health monitoring, and aging systems. Human Activity Recognition mainly leverages smartphones and wearable devices to collect sensory signals labeled with activity annotations and train machine learning models to recognize individuals' activity automatically. In order to deploy the Human Activity Recognition model in real-world scenarios, however, there are two major barriers. Firstly, sensor data and activity labels are traditionally collected using special experimental equipment in a controlled environment, which means fitting models trained with these datasets may result in poor generalization to real-life scenarios. Secondly, existing studies focus on single or a few modalities of sensor readings, which neglect useful information and its relations existing in multimodal sensor data. To tackle these issues, we propose a novel activity recognition model for multimodal sensory data fusion: Marfusion, and an experimental data collection platform for HAR tasks in real-world scenarios: MarSense. Specifically, Marfusion extensively uses a convolution structure to extract sensory features for each modality of the smartphone sensor and then fuse the multimodal features using the attention mechanism. MarSense can automatically collect a large amount of smartphone sensor data via smartphones among multiple users in their natural-used conditions and environment. To evaluate our proposed platform and model, we conduct a data collection experiment in real-life among university students and then compare our Marfusion model with several other state-of-the-art models on the collected datasets. Experimental Results do not only indicate that the proposed platform collected Human Activity Recognition data in the real-world scenario successfully, but also verify the advantages of the Marfusion model compared to existing models in Human Activity Recognition.

Multi-sensor Information Fusion Based on Machine Learning for Real Applications in Human Activity Recognition: State-of-the-art and Research Challenges.

Analysis of Multi-Sensor Fusion for Mobile and Wearable Sensor Based Human Activity Recognition

Multi-Sensor Image Fusion: A Survey of the State of the Art

Multi-Sensor Data Fusion for Accurate Human Activity Recognition with Deep Learning

Human activity recognition through deep learning: Leveraging unique and common feature fusion in wearable multi-sensor systems

A Multisensor Multiclassifier Hierarchical Fusion Model Based on Entropy Weight for Human Activity Recognition Using Wearable Inertial Sensors.

A Review on Multisensor Data Fusion for Wearable Health Monitoring

Deepfusion: A Deep Learning Framework For The Fusion Of Heterogeneous Sensory Data

A Multiscale Cross-Modal Interactive Fusion Network for Human Activity Recognition Using Wearable Sensors and Smartphones

Human activity recognition by fusing multiple sensor nodes in the wearable sensor systems

Imaging and Fusing Time Series for Wearable Sensor-Based Human Activity Recognition

Attention-Based Sensor Fusion for Human Activity Recognition Using IMU Signals

Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning

Marfusion: An Attention-Based Multimodal Fusion Model for Human Activity Recognition in Real-World Scenarios

Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Multi-Sensor Fusion Based Robot Self-Activity Recognition

Wearable Multi-Sensor Data Fusion Approach for Human Activity Recognition Using Machine Learning Algorithms

Unimodal and Multimodal Sensor Fusion for Wearable Activity Recognition

Technical Correspondence

Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity Recognition