Abstract:Human activity recognition (HAR) enables real-time monitoring of human movement, posture, and activity level, and can provide valuable information for health management. With the continuous advancement of Internet of Things (IoT) technology, wearable sensors and smartphones equipped with various types of sensors have become widely utilized to collect multimodal data for HAR. However, in multimodal HAR, current fusion methods fall short in capturing inter-modality correlations, hampering the full exploitation of complementary information between modalities and leading to lower recognition accuracy. We thus propose a novel multiscale cross-modal interactive fusion network (MCIFN), which can fully capture correlations between various modalities and obtain an effective fused representation for HAR. Specifically, we employ a multiscale parallel convolution module to extract features from each modality at multiple scales. Then, an interactive fusion strategy based on the cross-modal attention mechanism is introduced to adjust and enhance each modality based on its correlations with other modalities. Additionally, to resolve the information redundancy caused by the interactive fusion strategy, we utilize a hybrid attention module to focus on important information in the fusion representation. Extensive experiments conducted on three publicly available datasets and one private dataset demonstrate that our proposed network outperforms the previous baseline networks for HAR. Additionally, our proposed fusion strategy yielded a notable improvement in accuracy ranging from 1.87% to 9.96% compared to existing strategies. These findings imply that our newly proposed network can realize comprehensive multimodal fusion and effectively enhance HAR accuracy, potentially contributing to advancements in individual health management and personalized healthcare interventions.

Towards a Dynamic Inter-Sensor Correlations Learning Framework for Multi-Sensor-Based Wearable Human Activity Recognition.

Wearable Sensor Based Multimodal Human Activity Recognition Exploiting the Diversity of Classifier Ensemble.

A multi-graph convolutional network based wearable human activity recognition method using multi-sensors

Privacy-preserving Activity Recognition Using Multimodal Sensors in Smart Office

A Multi-dimensional Parallel Convolutional Connected Network Based on Multi-source and Multi-modal Sensor Data for Human Activity Recognition

Continual learning in sensor-based human activity recognition: An empirical benchmark analysis

Temporal-Spatial Dynamic Convolutional Neural Network for Human Activity Recognition Using Wearable Sensors

Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity Recognition

A Multidimensional Parallel Convolutional Connected Network Based on Multisource and Multimodal Sensor Data for Human Activity Recognition

Wearable Multi-Sensor Data Fusion Approach for Human Activity Recognition Using Machine Learning Algorithms

Deep Wavelet Convolutional Neural Networks for Multimodal Human Activity Recognition Using Wearable Inertial Sensors

Lifelong Learning in Sensor-Based Human Activity Recognition

A Novel Deep Multifeature Extraction Framework Based on Attention Mechanism Using Wearable Sensor Data for Human Activity Recognition

Multiscale Deep Feature Learning for Human Activity Recognition Using Wearable Sensors

Human activity recognition through deep learning: Leveraging unique and common feature fusion in wearable multi-sensor systems

DWOSC: Dynamic Weight Optimization and Smoothness Constraint for Sensor-Based Human Activity Recognition

A holistic multi-source transfer learning approach using wearable sensors for personalized daily activity recognition

SALIENCE: An Unsupervised User Adaptation Model for Multiple Wearable Sensors Based Human Activity Recognition

A Multiscale Cross-Modal Interactive Fusion Network for Human Activity Recognition Using Wearable Sensors and Smartphones

Disentangling Imperfect: A Wavelet-Infused Multilevel Heterogeneous Network for Human Activity Recognition in Flawed Wearable Sensor Data

Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition