Abstract:Recently, engagement has emerged as a key variable explaining the success of conversation. In the perspective of human-machine interaction, an automatic assessment of engagement becomes crucial to better understand the dynamics of an interaction and to design socially-aware robots. This paper presents a predictive model of the level of engagement in conversations. It shows in particular the interest of using a rich multimodal set of features, outperforming the existing models in this domain. In terms of methodology, study is based on two audio-visual corpora of naturalistic face-to-face interactions. These resources have been enriched with various annotations of verbal and nonverbal behaviors, such as smiles, head nods, and feedbacks. In addition, we manually annotated gestures intensity. Based on a review of previous works in psychology and human-machine interaction, we propose a new definition of the notion of engagement, adequate for the description of this phenomenon both in natural and mediated environments. This definition have been implemented in our annotation scheme. In our work, engagement is studied at the turn level, known to be crucial for the organization of the conversation. Even though there is still a lack of consensus around their precise definition, we have developed a turn detection tool. A multimodal characterization of engagement is performed using a multi-level classification of turns. We claim a set of multimodal cues, involving prosodic, mimo-gestural and morpho-syntactic information, is relevant to characterize the level of engagement of speakers in conversation. Our results significantly outperform the baseline and reach state-of-the-art level (0.76 weighted F-score). The most contributing modalities are identified by testing the performance of a two-layer perceptron when trained on unimodal feature sets and on combinations of two to four modalities. These results support our claim about multimodality: combining features related to the speech fundamental frequency and energy with mimo-gestural features leads to the best performance.

Can Prediction of Turn-management Willingness Improve Turn-changing Modeling?

Trimodal Prediction of Speaking and Listening Willingness to Help Improve Turn-Changing Modeling

Multimodal and Multitask Approach to Listener's Backchannel Prediction: Can Prediction of Turn-changing and Turn-management Willingness Improve Backchannel Modeling?

Response-conditioned Turn-taking Prediction

ML-SPEAK: A Theory-Guided Machine Learning Method for Studying and Predicting Conversational Turn-taking Patterns

Real-Time Multimodal Turn-taking Prediction to Enhance Cooperative Dialogue during Human-Agent Interaction

How to Represent Context Better? an Empirical Study on Context Modeling for Multi-turn Response Selection.

Early Turn-taking Prediction with Spiking Neural Networks for Human Robot Collaboration

How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models

Conversation Modeling to Predict Derailment

Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis

Predicting and Understanding Turn-Taking Behavior in Open-Ended Group Activities in Virtual Reality

Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions

Multilingual Turn-taking Prediction Using Voice Activity Projection

Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue

A multimodal approach for modeling engagement in conversation

Considering Temporal Connection Between Turns for Conversational Speech Synthesis

Can Turn-Taking Highlight the Nature of Non-Verbal Behavior: A Case Study

Predicting Evoked Emotions in Conversations

Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

Topic-Aware Multi-turn Dialogue Modeling