Abstract:Multi-modal bio-sensing has recently been used as effective research tools in affective computing, autism, clinical disorders, and virtual reality among other areas. However, none of the existing bio-sensing systems support multi-modality in a wearable manner outside well-controlled laboratory environments with research-grade measurements. This paper attempts to bridge this gap by developing a wearable multi-modal bio-sensing system capable of collecting, synchronizing, recording, and transmitting data from multiple bio-sensors: PPG, EEG, eye-gaze headset, body motion capture, GSR, etc., while also providing task modulation features including visual-stimulus tagging. This study describes the development and integration of various components of our system. We evaluate the developed sensors by comparing their measurements to those obtained by a standard research-grade bio-sensors. We first evaluate different sensor modalities of our headset, namely, earlobe-based PPG module with motion-noise canceling for ECG during heart-beat calculation. We also compare the steady-state visually evoked potentials measured by our shielded dry EEG sensors with the potentials obtained by commercially available dry EEG sensors. We also investigate the effect of head movements on the accuracy and precision of our wearable eye-gaze system. Furthermore, we carry out two practical tasks to demonstrate the applications of using multiple sensor modalities for exploring previously unanswerable questions in bio-sensing. Specifically, utilizing bio-sensing, we show which strategy works best for playing "Where is Waldo?" visual-search game, changes in EEG corresponding to true vs. false target fixations in this game, and predicting the loss/draw/win states through bio-sensing modalities while learning their limitations in a "Rock-Paper-Scissors" game.

Multi-modality Sensor Data Classification with Selective Attention

Wearable Sensor Based Multimodal Human Activity Recognition Exploiting the Diversity of Classifier Ensemble.

A Wearable Multi-Modal Bio-Sensing System Towards Real-World Applications

MultiSense: Cross Labelling and Learning Human Activities Using Multimodal Sensing Data

An Automatic Cardiac Arrhythmia Classification System with Wearable Electrocardiogram

A Semisupervised Recurrent Convolutional Attention Model for Human Activity Recognition

Interpretable Parallel Recurrent Neural Networks with Convolutional Attentions for Multi-Modality Activity Modeling

AttnSense: Multi-level Attention Mechanism for Multimodal Human Activity Recognition

Real-time activity classification using ambient and wearable sensors.

SALIENCE: An Unsupervised User Adaptation Model for Multiple Wearable Sensors Based Human Activity Recognition

A Novel Deep Multifeature Extraction Framework Based on Attention Mechanism Using Wearable Sensor Data for Human Activity Recognition

Advances in Machine Learning for Wearable Sensors

A Survey on Multimodal Wearable Sensor-based Human Action Recognition

Recognizing Multi-User Activities Using Wearable Sensors in a Smart Home

Multimodal sensor data fusion for in-situ classification of animal behavior using accelerometry and GNSS data

Multimodal Data Processing Framework for Smart City: A Positional-Attention Based Deep Learning Approach.

A Boundary Consistency-aware Multi-task Learning Framework for Joint Activity Segmentation and Recognition with Wearable Sensors

Gait-based Identification Using Wearable Multimodal Sensing and Attention Neural Networks

Vision-Sensor Attention Based Continual Multimodal Egocentric Activity Recognition.

Hierarchical Multi-Classification for Sensor-based Badminton Activity Recognition

Bi-STAN: bilinear spatial-temporal attention network for wearable human activity recognition