Abstract:This paper addresses learning and recognition of human behavior models from multimodal observation in a smart home environment. The proposed approach is part of a framework for acquiring a high-level contextual model for human behavior in an augmented environment. A 3-D video tracking system creates and tracks entities (persons) in the scene. Further, a speech activity detector analyzes audio streams coming from head set microphones and determines for each entity, whether the entity speaks or not. An ambient sound detector detects noises in the environment. An individual role detector derives basic activity like “walking” or “interacting with table” from the extracted entity properties of the 3-D tracker. from the derived multimodal observations, different situations like “aperitif” or “presentation” are learned and detected using statistical models (HMMs). The objective of the proposed general framework is two-fold: the automatic offline analysis of human behavior recordings and the online detection of learned human behavior models. To evaluate the proposed approach, several multimodal recordings showing different situations have been conducted. The obtained results, in particular for offline analysis, are very good, showing that multimodality as well as multiperson observation generation are beneficial for situation recognition. Note to Practitioners—This paper was motivated by the problem of automatically recognizing human behavior and interactions in a smart home environment. The smart home environment is equipped with cameras and microphones that permit the observation of human activity in the scene. The objective is first to visualize the perceived human activities (e.g., for videoconferencing or surveillance of elderly people), and then to provide appropriate services based on these activities. We adopt a layered approach for human activity recognition in the environment. The layered framework is motivated by the human perception of human behavior in the scene (white box). The system first recognizes basic activities of individuals, called roles, like “interacting with table” or “walking.” Then, based on the recognized individual roles, group situations like “aperitif,” “presentation,” or “siesta” are recognized. in this paper, we describe an implementation that is based on a 3-D video tracking system, as well as speech activity detection using head set microphones. We evaluated the system for offline (a posteriori) situation classification and online (in scenario) situation recognition. A prototype system has been realized and installed at France Télécom R&D, visualizing current human behavior in the smart home to a distant user using a web interface. An open issue is still the detection of group dynamics and group formation, which is necessary for group situation recognition in (informal) real settings.

Continuous Multi-Modal Human Interest Detection for a Domestic Companion Humanoid Robot.

Implementation of Engagement Detection for Human–Robot Interaction in Complex Environments

A Multimodal Face Detection System for Elderly Companion Robot

Human–robot non-verbal interaction empowered by real-time auditory and visual multiple-talker tracking

A Multimodal Emotional Communication Based Humans-Robots Interaction System

A multimodal domestic service robot interaction system for people with declined abilities to express themselves

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Contactless Interaction System Based on Facial Expression Recognition for Humanoid Piano Robot

Multimodal Approach to Human-Face Detection and Tracking

Vision- and tactile-based continuous multimodal intention and attention recognition for safer physical human-robot interaction

Intelligent Facial Emotion Recognition and Semantic-Based Topic Detection for A Humanoid Robot

Robot System Assistant (RoSA): Towards Intuitive Multi-Modal and Multi-Device Human-Robot Interaction

Intelligent communication of two humanoid robots based on computer vison

Multi-Class Detection and Segmentation of Objects in Depth

Detecting Human Behavior Models From Multimodal Observation in a Smart Home

Robot recognizing humans intention and interacting with humans based on a multi-task model combining ST-GCN-LSTM model and YOLO model

Robot Awareness of Human Behaviors and Interacting with Humans by A Two-Step Identifying Approach combining LSTM and SSD.

ROBUST REAL-TIME MULTI-FACE DETECTION FOR INTELLIGENTHUMAN-ROBOT INTERACTION

Multi-Sensor Context-Aware Based Chatbot Model: An Application of Humanoid Companion Robot

Human-Robot Interaction using Intention Recognition

A Novel Continuous Learning and Collaborative Decision Making Mechanism for Real-Time Cooperation of Humanoid Service Robots