Abstract:Human activity recognition (HAR) based on wearable sensors has been widely used in various fields such as health monitoring, healthcare, and fitness due to its portability, accuracy, and real-time capabilities. Currently, advanced technologies involve converting time-series into images and deep learning for recognition, which addresses issues of subjectivity and dependence on data quality inherent in traditional processing methods. However, each of the current methods for converting time-series data into images only focuses on one type of feature representation such that one of these methods is insufficient to fully characterize the data, which results in lower accuracy in activity recognition. To address the above problems, we propose a novel method for cross-modal image representation of time-series and a dual-stream feature enhanced network model, which enables HAR based on a single-node wearable sensor. Firstly, three methods including Markov Transition Field (MTF), Recurrence Plot (RP), and Gramian Angular Field (GAF), are employed to encode the time-series into three channels (R, G, B) of color images, facilitating the fusion of features such as amplitude variation, nonlinearity, and local temporal relationship. Then, a multi-channel CNN with a Global Attention Mechanism (GAM) is employed in the model to process images, which captures both channel and spatial information. A network combining CNN and Long Short-Term Memory (LSTM) with Self-Attention Mechanism (SA) is utilized to process time-series, which captures temporal features. Simultaneously, the introduction of residual structures makes the network easy to train, thereby enhancing overall performance. The experimental results demonstrate that the proposed model can accurately identify six different gymnastic activities, achieving an accuracy rate of 99.40%. This study provides a new direction for processing time-series and offers better applications in the field of HAR based on wearable sensors.

Human Activity Recognition Based on Multi-Modal Fusion

Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition

Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Attention-Based Sensor Fusion for Human Activity Recognition Using IMU Signals

Privacy-preserving Activity Recognition Using Multimodal Sensors in Smart Office

A Multimodal Fusion Approach for Human Activity Recognition

A Multiscale Cross-Modal Interactive Fusion Network for Human Activity Recognition Using Wearable Sensors and Smartphones

Human activity recognition through deep learning: Leveraging unique and common feature fusion in wearable multi-sensor systems

Weighted Fusion of Multiple Classifiers for Human Activity Recognition

CIR-DFENet: Incorporating Cross-Modal Image Representation and Dual-Stream Feature Enhanced Network for Activity Recognition

Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things

Multi-resolution Fusion Convolutional Network for Open Set Human Activity Recognition

A Multi-dimensional Parallel Convolutional Connected Network Based on Multi-source and Multi-modal Sensor Data for Human Activity Recognition

A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors

Enhancing Human Activity Recognition through Integrated Multimodal Analysis: A Focus on RGB Imaging, Skeletal Tracking, and Pose Estimation

Human Activity Recognition Method Based on FMCW Radar Sensor with Multi-Domain Feature Attention Fusion Network

A Comprehensive Methodological Survey of Human Activity Recognition Across Divers Data Modalities

Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity Recognition

A multi-scale feature extraction fusion model for human activity recognition

A Multidimensional Parallel Convolutional Connected Network Based on Multisource and Multimodal Sensor Data for Human Activity Recognition

Multihead-Res-SE Residual Network with Attention for Human Activity Recognition