Abstract:Numerous methods and applications have been proposed in human activity recognition (HAR). This paper presents a mini-survey of recent HAR studies and our originally developed benchmark datasets of two types using environmental sensors. For the first dataset, we specifically examine human pose estimation and slight motion recognition related to activities of daily living (ADL). Our proposed method employs OpenPose. It describes feature vectors without effects of objects or scene features, but with a convolutional neural network (CNN) with the VGG-16 backbone, which recognizes behavior patterns after classifying the obtained images into learning and verification subsets. The first dataset comprises time-series panoramic images obtained using a fisheye lens monocular camera with a wide field of view. We attempted to recognize five behavior patterns: eating, reading, operating a smartphone, operating a laptop computer, and sitting. Even when using panoramic images including distortions, results demonstrate the capability of recognizing properties and characteristics of slight motions and pose-based behavioral patterns. The second dataset was obtained using five environmental sensors: a thermopile sensor, a CO2 sensor, and air pressure, humidity, and temperature sensors. Our proposed sensor system obviates the need for constraint; it also preserves each subject’s privacy. Using a long short-term memory (LSTM) network combined with CNN, which is a deep-learning model dealing with time-series features, we recognized eight behavior patterns: eating, operating a laptop computer, operating a smartphone, playing a game, reading, exiting, taking a nap, and sitting. The recognition accuracy for the second dataset was lower than for the first dataset consisting of images, but we demonstrated recognition of behavior patterns from time-series of weak sensor signals. The recognition results for the first dataset, after accuracy evaluation, can be reused for automatically annotated labels applied to the second dataset. Our proposed method actualizes semi-automatic annotation, false recognized category detection, and sensor calibration. Feasibility study results show the new possibility of HAR used for ADL based on unique sensors of two types.

Real‐world multimodal lifelog dataset for human behavior study

Towards a Multimodal Human Activity Dataset for Healthcare

Multi-Modal Data Collection for Measuring Health, Behavior, and Living Environment of Large-Scale Participant Cohorts: Conceptual Framework and Findings from Deployments

An Activity Management System for Office Workers Using Multimodal Data

Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning

Know Yourself: Physical and Psychological Self-Awareness With Lifelog

Recognizing Human Daily Activity Using Social Media Sensors And Deep Learning

A Public Domain Dataset for Human Activity Recognition in Free-Living Conditions

Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning

A Mini-Survey and Feasibility Study of Deep-Learning-Based Human Activity Recognition from Slight Feature Signals Obtained Using Privacy-Aware Environmental Sensors

Multimodal data for behavioural authentication in Internet of Things environments

A multimodal physiological dataset for driving behaviour analysis

A multimodal fMRI dataset unifying naturalistic processes with a rich array of experimental tasks

Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving

GLOBEM Dataset: Multi-Year Datasets for Longitudinal Human Behavior Modeling Generalization

MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing

Multimodal Group Activity Dataset for Classroom Engagement Level Prediction

A real-world dataset of group emotion experiences based on physiological data

CAPTURE-24: A large dataset of wrist-worn activity tracker data collected in the wild for human activity recognition

Multimodal video and IMU kinematic dataset on daily life activities using affordable devices (VIDIMU)