Abstract:The combination of increased life expectancy and falling birth rates is resulting in an aging population. Wearable Sensor-based Human Activity Recognition (WSHAR) emerges as a promising assistive technology to support the daily lives of older individuals, unlocking vast potential for human-centric applications. However, recent surveys in WSHAR have been limited, focusing either solely on deep learning approaches or on a single sensor modality. In real life, our human interact with the world in a multi-sensory way, where diverse information sources are intricately processed and interpreted to accomplish a complex and unified sensing system. To give machines similar intelligence, multimodal machine learning, which merges data from various sources, has become a popular research area with recent advancements. In this study, we present a comprehensive survey from a novel perspective on how to leverage multimodal learning to WSHAR domain for newcomers and researchers. We begin by presenting the recent sensor modalities as well as deep learning approaches in HAR. Subsequently, we explore the techniques used in present multimodal systems for WSHAR. This includes inter-multimodal systems which utilize sensor modalities from both visual and non-visual systems and intra-multimodal systems that simply take modalities from non-visual systems. After that, we focus on current multimodal learning approaches that have applied to solve some of the challenges existing in WSHAR. Specifically, we make extra efforts by connecting the existing multimodal literature from other domains, such as computer vision and natural language processing, with current WSHAR area. Finally, we identify the corresponding challenges and potential research direction in current WSHAR area for further improvement.

Application of Transfer Learning Approaches in Multimodal Wearable Human Activity Recognition

Wearable Sensor Based Multimodal Human Activity Recognition Exploiting the Diversity of Classifier Ensemble.

Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition

A holistic multi-source transfer learning approach using wearable sensors for personalized daily activity recognition

Weakly Supervised Multi-Task Representation Learning for Human Activity Analysis Using Wearables

Transfer Learning Model Knowledge Across Multi-sensors Locations over Body Sensor Network

SMTDKD: A Semantic-Aware Multimodal Transformer Fusion Decoupled Knowledge Distillation Method for Action Recognition

A Survey on Multimodal Wearable Sensor-based Human Action Recognition

Transfer Learning for Activity Recognition in Mobile Health

Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables

A Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognition

Ensembles of Deep LSTM Learners for Activity Recognition using Wearables

Wearable Sensors for Activity Recognition in Ultimate Frisbee Using Convolutional Neural Networks and Transfer Learning

A Semisupervised Recurrent Convolutional Attention Model for Human Activity Recognition

Deep Transfer Learning with Graph Neural Network for Sensor-Based Human Activity Recognition

Deep learning based multimodal complex human activity recognition using wearable devices

Recognizing Multi-User Activities Using Wearable Sensors in a Smart Home

Disentangling Imperfect: A Wavelet-Infused Multilevel Heterogeneous Network for Human Activity Recognition in Flawed Wearable Sensor Data

Few-Shot Learning-Based Human Activity Recognition

Transfer Learning for Human Activity Recognition using Representational Analysis of Neural Networks

AROMA: A Deep Multi-Task Learning Based Simple and Complex Human Activity Recognition Method Using Wearable Sensors