Abstract:Extended reality (XR) systems are about to be integrated into our daily lives and will provide support in a variety of fields such as education and coaching. Enhancing user experience demands agents that are capable of displaying realistic affective and social behaviors within these systems, and, as a prerequisite, with the capability of understanding their interaction partner and responding appropriately. Based on our literature review of recent works published in the field of co-speech gesture generation, researchers have developed complex models capable of generating gestures characterized by a high level of human-likeness and speaker appropriateness. Nevertheless, this is only true in settings where the agent has an active status (i.e., the agent acts as the speaker), or it is delivering a monologue in a non-interactive setting. However, as illustrated in multiple works and competitions like the GENEA Challenge, these models remain inadequate in generating interlocutor-aware gestures. We consider interlocutor-aware gesture generation the process of displaying gestures that take into account the conversation partner's behavior. Moreover, in settings where the agent is the listener, generated gestures lack the level of naturalness that we expect from a face-to-face conversation. To overcome these issues, we have designed a pipeline, called TAG2G, composed of a diffusion model, which was demonstrated to be a stable and powerful tool in gesture generation, and a vector-quantized variational auto-encoder (VQVAE), widely employed to produce meaningful gesture embeddings. Refocusing from monadic to dyadic multimodal input settings (i.e., taking into account text, audio, and previous gestures of both participants of a conversation) allows us to explore and infer the complex interaction mechanisms that lie in a balanced two-sided conversation. As per our results, a multi-agent conversational input setup improves the generated gestures' appropriateness with respect to the conversational counterparts. Conversely, when the agent is speaking, a monadic approach performs better in terms of the generated gestures' appropriateness in relation to the speech.

DiffuGesture: Generating Human Gesture from Two-person Dialogue with Diffusion Models

The DiffuseStyleGesture+ entry to the GENEA Challenge 2023

Multimodal-driven Talking Face Generation, Face Swapping, Diffusion Model

Conversational Co-Speech Gesture Generation via Modeling Dialog Intention, Emotion, and Context with Diffusion Models

Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation

Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation

DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model

DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation

ExpGest: Expressive Speaker Generation Using Diffusion Model and Hybrid Audio-Text Guidance

DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models

EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model

InterGen: Diffusion-Based Multi-human Motion Generation Under Complex Interactions

ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis

TAG2G: A Diffusion-Based Approach to Interlocutor-Aware Co-Speech Gesture Generation

C2G2: Controllable Co-speech Gesture Generation with Latent Diffusion Model

A Unified Editing Method for Co-Speech Gesture Generation via Diffusion Inversion

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework

The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settings