Abstract:During face-to-face interactions, people naturally integrate nonverbal behaviors such as facial expressions and body postures as part of the conversation to infer the communicative intent or emotional state of their interlocutor. The interpretation of these nonverbal behaviors will often be contextualized by interactional cues such as the previous spoken question, the general discussion topic or the physical environment. A critical step in creating computers able to understand or participate in this type of social face-to-face interactions is to develop a computational platform to synchronously recognize nonverbal behaviors as part of the interactional context. In this platform, information for the acoustic and visual modalities should be carefully synchronized and rapidly processed. At the same time, contextual and interactional cues should be remembered and integrated to better interpret nonverbal (and verbal) behaviors. In this article, we introduce a real-time computational framework, MultiSense, which offers flexible and efficient synchronization approaches for context-based nonverbal behavior analysis. MultiSense is designed to utilize interactional cues from both interlocutors (e.g., from the computer and the human participant) and integrate this contextual information when interpreting nonverbal behaviors. MultiSense can also assimilate behaviors over a full interaction and summarize the observed affective states of the user. We demonstrate the capabilities of the new framework with a concrete use case from the mental health domain where MultiSense is used as part of a decision support tool to assess indicators of psychological distress such as depression and post-traumatic stress disorder (PTSD). In this scenario, MultiSense not only infers psychological distress indicators from nonverbal behaviors but also broadcasts the user state in real-time to a virtual agent (i.e., a digital interviewer) designed to conduct semi-structured interviews with human participants. Our experiments show the added value of our multimodal synchronization approaches and also demonstrate the importance of MultiSense contextual interpretation when inferring distress indicators.

Human nonverbal behavior multi-sourced ontological annotation

Using a Kinect Sensor to Acquire Biological Motion: Toolbox and Evaluation.

Web-based visualisation of head pose and facial expressions changes: monitoring human activity using depth data

An Empirical Study for Human Behavior Analysis

Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey

Nonverbal Interaction Detection

Using Computer Vision to Annotate Video-Recoded Direct Observation of Physical Behavior

Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions

Visual Inference of Human Emotion and Behaviour

Human-Robot Dialogue Annotation for Multi-Modal Common Ground

HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling

MultiSense—Context-Aware Nonverbal Behavior Analysis Framework: A Psychological Distress Use Case

Headset: Human emotion awareness under partial occlusions multimodal dataset

Understanding the Roles of Video and Sensor Data in the Annotation of Human Activities

Towards Enhanced Human Activity Recognition through Natural Language Generation and Pose Estimation

A survey on ontologies for human behavior recognition

BEHAVE: Dataset and Method for Tracking Human Object Interactions

Automated Data Annotation for 6-DoF AI-Based Navigation Algorithm Development

Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenes

A Probabilistic Semi-Supervised Approach to Multi-Task Human Activity Modeling

AN ALGORITHM FOR DETECTING NON-VERBAL MARKERS OF HUMAN BEHAVIOR ON VIDEO