Abstract:Numerous methods and applications have been proposed in human activity recognition (HAR). This paper presents a mini-survey of recent HAR studies and our originally developed benchmark datasets of two types using environmental sensors. For the first dataset, we specifically examine human pose estimation and slight motion recognition related to activities of daily living (ADL). Our proposed method employs OpenPose. It describes feature vectors without effects of objects or scene features, but with a convolutional neural network (CNN) with the VGG-16 backbone, which recognizes behavior patterns after classifying the obtained images into learning and verification subsets. The first dataset comprises time-series panoramic images obtained using a fisheye lens monocular camera with a wide field of view. We attempted to recognize five behavior patterns: eating, reading, operating a smartphone, operating a laptop computer, and sitting. Even when using panoramic images including distortions, results demonstrate the capability of recognizing properties and characteristics of slight motions and pose-based behavioral patterns. The second dataset was obtained using five environmental sensors: a thermopile sensor, a CO2 sensor, and air pressure, humidity, and temperature sensors. Our proposed sensor system obviates the need for constraint; it also preserves each subject’s privacy. Using a long short-term memory (LSTM) network combined with CNN, which is a deep-learning model dealing with time-series features, we recognized eight behavior patterns: eating, operating a laptop computer, operating a smartphone, playing a game, reading, exiting, taking a nap, and sitting. The recognition accuracy for the second dataset was lower than for the first dataset consisting of images, but we demonstrated recognition of behavior patterns from time-series of weak sensor signals. The recognition results for the first dataset, after accuracy evaluation, can be reused for automatically annotated labels applied to the second dataset. Our proposed method actualizes semi-automatic annotation, false recognized category detection, and sensor calibration. Feasibility study results show the new possibility of HAR used for ADL based on unique sensors of two types.

Understanding the Roles of Video and Sensor Data in the Annotation of Human Activities

Continual learning in sensor-based human activity recognition: An empirical benchmark analysis

Online Continual Learning for Human Activity Recognition

Using Computer Vision to Annotate Video-Recoded Direct Observation of Physical Behavior

SLearn: Shared Learning Human Activity Labels Across Multiple Datasets.

An Empirical Study for Human Behavior Analysis

An active semi-supervised deep learning model for human activity recognition

A Comprehensive Methodological Survey of Human Activity Recognition Across Divers Data Modalities

A Matter of Annotation: An Empirical Study on In Situ and Self-Recall Activity Annotations from Wearable Sensors

A Semi-Automatic Annotation Approach for Human Activity Recognition

Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams

A Mini-Survey and Feasibility Study of Deep-Learning-Based Human Activity Recognition from Slight Feature Signals Obtained Using Privacy-Aware Environmental Sensors

Augmented Adversarial Learning for Human Activity Recognition with Partial Sensor Sets

Video2IMU: Realistic IMU features and signals from videos

Comprehensive machine and deep learning analysis of sensor-based human activity recognition

Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Them

The State-of-the-Art Sensing Techniques in Human Activity Recognition: A Survey

Semi-Supervised Adversarial Learning Using LSTM for Human Activity Recognition

Overview of Human Activity Recognition Using Sensor Data

Vi2ACT:Video-enhanced Cross-modal Co-learning with Representation Conditional Discriminator for Few-shot Human Activity Recognition

Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning