Exploring Multimodal Nonverbal Functional Features for Predicting the Subjective Impressions of Interlocutors
Koya Ito,Yoko Ishii,Ryo Ishii,Shin-Ichiro Eitoku,Kazuhiro Otsuka
DOI: https://doi.org/10.1109/access.2024.3426537
IF: 3.9
2024-07-19
IEEE Access
Abstract:This paper proposes models for predicting the subjective impressions of interlocutors in discussions according to multimodal nonverbal behaviors. To that end, we focus mainly on the functional aspects of head movement and facial expressions as insightful cues. For example, head movement functions include the speaker's rhythm and the listener's back channel and thinking processes, as well as their positive emotions. Facial expression functions include emotional expressions and communicative functions such as the speaker addressing the listener and the listener's affirmation. In addition, our model employs synergetic functions, which are jointly performed with head movements and facial expressions, assuming that the simultaneous appearance of head and face functions could strengthen the results or lead to multiplexing. On the basis of these nonverbal functions, we define a set of functional features, including the rate of occurrence and composition balance among different functions that emerge during conversation. Then, a feature selection scheme is used to identify the best combinations of intermodal and intramodal features. In the experiments, an SA-Off corpus of 17 groups of discussions involving 4 female participants was used, including interlocutors' self-reported scores for 16 impression items felt during the discussion, such as helpfulness and interest. The experiments confirmed that our models' predictions were significantly correlated with the self-reported scores for more than 70% of the impression items. These results indicate the effectiveness of multimodal nonverbal functional features for predicting subjective impressions.
computer science, information systems,telecommunications,engineering, electrical & electronic