Abstract:Non-verbal behavior is a central challenge in understanding the dynamics of a conversation and the affective states between interlocutors arising from the interaction. Although psychological research has demonstrated that non-verbal behaviors vary across cultures, limited computational analysis has been conducted to clarify these differences and assess their impact on engagement recognition. To gain a greater understanding of engagement and non-verbal behaviors among a wide range of cultures and language spheres, in this study we conduct a multilingual computational analysis of non-verbal features and investigate their role in engagement and engagement prediction. To achieve this goal, we first expanded the NoXi dataset, which contains interaction data from participants living in France, Germany, and the United Kingdom, by collecting session data of dyadic conversations in Japanese and Chinese, resulting in the enhanced dataset NoXi+J. Next, we extracted multimodal non-verbal features, including speech acoustics, facial expressions, backchanneling and gestures, via various pattern recognition techniques and algorithms. Then, we conducted a statistical analysis of listening behaviors and backchannel patterns to identify culturally dependent and independent features in each language and common features among multiple languages. These features were also correlated with the engagement shown by the interlocutors. Finally, we analyzed the influence of cultural differences in the input features of LSTM models trained to predict engagement for five language datasets. A SHAP analysis combined with transfer learning confirmed a considerable correlation between the importance of input features for a language set and the significant cultural characteristics analyzed.

Cross-Discourse and Multilingual Exploration of Textual Corpora with the DualNeighbors Algorithm

Cross-Reading by Leveraging a Hybrid Index of Heterogeneous Information.

How We Do Things With Words: Analyzing Text as Social and Cultural Data

Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement

Comprehending Lexical and Affective Ontologies in the Demographically Diverse Spatial Social Media Discourse

Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective

Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora

Scalable Community Discovery on Textual Data with Relations

Where New Words Are Born: Distributional Semantic Analysis of Neologisms and Their Semantic Neighborhoods

The Geometry of Culture: Analyzing Meaning through Word Embeddings

Graph-Community Detection for Cross-Document Topic Segment Relationship Identification

Expanding the Text Classification Toolbox with Cross-Lingual Embeddings

Emerging Language Spaces Learned From Massively Multilingual Corpora

WMDecompose: A Framework for Leveraging the Interpretable Properties of Word Mover's Distance in Sociocultural Analysis

Mining Asymmetric Intertextuality

Modeling Socio-Cultural Phenomena in Discourse

Cultural cartography with word embeddings

Parallel Corpus Research and Target Language Representativeness: The Contrastive, Typological, and Translation Mining Traditions

Semantic Word Clouds with Background Corpus Normalization and t-distributed Stochastic Neighbor Embedding

Dual Relation Network for Scene Text Recognition

Challenges and Strategies in Cross-Cultural NLP