Abstract:Human activity recognition (HAR) enables real-time monitoring of human movement, posture, and activity level, and can provide valuable information for health management. With the continuous advancement of Internet of Things (IoT) technology, wearable sensors and smartphones equipped with various types of sensors have become widely utilized to collect multimodal data for HAR. However, in multimodal HAR, current fusion methods fall short in capturing inter-modality correlations, hampering the full exploitation of complementary information between modalities and leading to lower recognition accuracy. We thus propose a novel multiscale cross-modal interactive fusion network (MCIFN), which can fully capture correlations between various modalities and obtain an effective fused representation for HAR. Specifically, we employ a multiscale parallel convolution module to extract features from each modality at multiple scales. Then, an interactive fusion strategy based on the cross-modal attention mechanism is introduced to adjust and enhance each modality based on its correlations with other modalities. Additionally, to resolve the information redundancy caused by the interactive fusion strategy, we utilize a hybrid attention module to focus on important information in the fusion representation. Extensive experiments conducted on three publicly available datasets and one private dataset demonstrate that our proposed network outperforms the previous baseline networks for HAR. Additionally, our proposed fusion strategy yielded a notable improvement in accuracy ranging from 1.87% to 9.96% compared to existing strategies. These findings imply that our newly proposed network can realize comprehensive multimodal fusion and effectively enhance HAR accuracy, potentially contributing to advancements in individual health management and personalized healthcare interventions.

Towards Pervasive Sensing

Towards Pervasive Sensing: A Multimodal Approach Via CSI and RGB Image Modalities Fusion

Privacy-preserving Activity Recognition Using Multimodal Sensors in Smart Office

Multi-Sensor Image Fusion: A Survey of the State of the Art

Human Activity Recognition Based on Multi-Modal Fusion

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition

A Multiscale Cross-Modal Interactive Fusion Network for Human Activity Recognition Using Wearable Sensors and Smartphones

CHA-Sens: an End-to-End Comprehensive Residual Convolution Framework for CSI-based Human Activity Sensing.

Perceptual-Based Fusion Of Ir And Visual Images For Human Detection

Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial Sensors

Multi-sensor Information Fusion Based on Machine Learning for Real Applications in Human Activity Recognition: State-of-the-art and Research Challenges.

The SES Framework and Frequency Domain Information Fusion Strategy for Human Activity Recognition

Enhancing Human Activity Recognition through Integrated Multimodal Analysis: A Focus on RGB Imaging, Skeletal Tracking, and Pose Estimation

Human-centric multimodal fusion network for robust action recognition

Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

CSFuser: A Cascade Siamese Fusion Architecture for RGB-Infrared Object Detection

Multiscale Channel Attention Network for Infrared and Visible Image Fusion.

A robust infrared and visible image fusion framework via multi-receptive-field attention and color visual perception

Using a Selective Ensemble Support Vector Machine to Fuse Multimodal Features for Human Action Recognition

Human activity recognition through deep learning: Leveraging unique and common feature fusion in wearable multi-sensor systems