Abstract:In networked virtual reality (VR), user behaviors, individual differences, and group dynamics can serve as important signals into future speech behaviors, such as who the next speaker will be and the timing of turn-taking behaviors. The ability to predict and understand these behaviors offers opportunities to provide adaptive and personalized assistance, for example helping users with varying sensory abilities navigate complex social scenes and instantiating virtual moderators with natural behaviors. In this work, we predict turn-taking behaviors using features extracted based on social dynamics literature. We discuss results from a large-scale VR classroom dataset consisting of 77 sessions and 1660 minutes of small-group social interactions collected over four weeks. In our evaluation, gradient boosting classifiers achieved the best performance, with accuracies of 0.71--0.78 AUC (area under the ROC curve) across three tasks concerning the "what", "who", and "when" of turn-taking behaviors. In interpreting these models, we found that group size, listener personality, speech-related behavior (e.g., time elapsed since the listener's last speech event), group gaze (e.g., how much the group looks at the speaker), as well as the listener's and previous speaker's head pitch, head y-axis position, and left hand y-axis position more saliently influenced predictions. Results suggested that these features remain reliable indicators in novel social VR settings, as prediction performance is robust over time and with groups and activities not used in the training dataset. We discuss theoretical and practical implications of the work.

Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection

Multilingual Turn-taking Prediction Using Voice Activity Projection

Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection

A Novel and Efficient Voice Activity Detector Using Shape Features of Speech Wave.

How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models

Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis

Real-time Architecture for Audio-Visual Active Speaker Detection.

Real-Time Multimodal Turn-taking Prediction to Enhance Cooperative Dialogue during Human-Agent Interaction

A cross-talk robust multichannel VAD model for multiparty agent interactions trained using synthetic re-recordings

Real-time Caller Intent Detection In Human-Human Customer Support Spoken Conversations

Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

Real-Time Causal Spectro-Temporal Voice Activity Detection Based on Convolutional Encoding and Residual Decoding

Semantic VAD: Low-Latency Voice Activity Detection for Speech Interaction

Voice Activity Detection Based on Time-Delay Neural Networks

Predicting and Understanding Turn-Taking Behavior in Open-Ended Group Activities in Virtual Reality

Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

End-to-End Speaker-Dependent Voice Activity Detection

A New Vad Framework Using Statistical Model And Human Knowledge Based Empirical Rule

IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities

VoCopilot: Voice-Activated Tracking of Everyday Interactions

Multi-task Joint-Learning for Robust Voice Activity Detection