Human Activity Recognition Algorithm with Physiological and Inertial Signals Fusion: Photoplethysmography, Electrodermal Activity, and Accelerometry

Justin Gilmore,Mona Nasseri

DOI: https://doi.org/10.3390/s24103005

IF: 3.9

2024-05-10

Sensors

Abstract:Inertial signals are the most widely used signals in human activity recognition (HAR) applications, and extensive research has been performed on developing HAR classifiers using accelerometer and gyroscope data. This study aimed to investigate the potential enhancement of HAR models through the fusion of biological signals with inertial signals. The classification of eight common low-, medium-, and high-intensity activities was assessed using machine learning (ML) algorithms, trained on accelerometer (ACC), blood volume pulse (BVP), and electrodermal activity (EDA) data obtained from a wrist-worn sensor. Two types of ML algorithms were employed: a random forest (RF) trained on features; and a pre-trained deep learning (DL) network (ResNet-18) trained on spectrogram images. Evaluation was conducted on both individual activities and more generalized activity groups, based on similar intensity. Results indicated that RF classifiers outperformed corresponding DL classifiers at both individual and grouped levels. However, the fusion of EDA and BVP signals with ACC data improved DL classifier performance compared to a baseline DL model with ACC-only data. The best performance was achieved by a classifier trained on a combination of ACC, EDA, and BVP images, yielding F1-scores of 69 and 87 for individual and grouped activity classifications, respectively. For DL models trained with additional biological signals, almost all individual activity classifications showed improvement (p-value < 0.05). In grouped activity classifications, DL model performance was enhanced for low- and medium-intensity activities. Exploring the classification of two specific activities, ascending/descending stairs and cycling, revealed significantly improved results using a DL model trained on combined ACC, BVP, and EDA spectrogram images (p-value < 0.05).

engineering, electrical & electronic,instruments & instrumentation,chemistry, analytical

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in human activity recognition (HAR), to improve the accuracy of activity recognition by fusing physiological signals (such as photoplethysmography [BVP] and electrodermal activity [EDA]) with inertial signals (accelerometer data). Traditional HAR mainly relies on accelerometer data, but these data may be insufficient when recognizing certain activities with low physical movement but high energy consumption. Therefore, this paper explores the combined use of BVP and EDA signals with accelerometer data to enhance the activity recognition performance of feature - based and deep - learning classifiers.

Specifically, the researchers used two methods, random forest (RF) and a pre - trained deep - learning network (ResNet - 18), to train and test the accelerometer data alone and the combination of accelerometer data with BVP and EDA signals respectively. The goal of the study was to evaluate whether these non - inertial signals could significantly improve the performance of HAR models, especially in recognizing specific activities such as going up and down stairs and cycling.

The main contributions of the paper are as follows:
1. **Fusing multiple physiological signals**: By fusing BVP and EDA signals, the researchers hope to overcome the limitations of a single inertial signal in certain activity recognition.
2. **Comparing different classifiers**: Comparing the performance of random forest and deep - learning models under fused signals, and evaluating which method is more effective for multi - modal data.
3. **Improvement in specific activities**: Paying special attention to the two activities of going up and down stairs and cycling, and verifying the improvement effect of fused signals on the recognition of these activities.

Through these studies, the paper aims to provide new ideas and technical solutions for the HAR field, especially in multi - modal signal fusion, and provide references for the design of future wearable devices and health monitoring systems.