Abstract:Modeling virtual agents with behavior style is one factor for personalizing human-agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero-shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive; while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of a speaker whose data are not part of the training phase, without requiring any further training or fine-tuning. The first goal of our model is to generate the gestures of a source speaker based on the content of two input modalities-Mel spectrogram and text semantics. The second goal is to condition the source speaker's predicted gestures on the multimodal behavior style embedding of a target speaker. The third goal is to allow zero-shot style transfer of speakers unseen during training without re-training the model. Our system consists of two main components: (1) a speaker style encoder network that learns to generate a fixed-dimensional speaker embedding style from a target speaker multimodal data (mel-spectrogram, pose, and text) and (2) a sequence-to-sequence synthesis network that synthesizes gestures based on the content of the input modalities-text and mel-spectrogram-of a source speaker and conditioned on the speaker style embedding. We evaluate that our model is able to synthesize gestures of a source speaker given the two input modalities and transfer the knowledge of target speaker style variability learned by the speaker style encoder to the gesture generation task in a zero-shot setup, indicating that the model has learned a high-quality speaker representation. We conduct objective and subjective evaluations to validate our approach and compare it with baselines.

GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech

StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis

GenerTTS: Pronunciation Disentanglement for Timbre and Style Generalization in Cross-Lingual Text-to-Speech

StyleS2ST: Zero-shot Style Transfer for Direct Speech-to-speech Translation

StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion

Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer

StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis

U-Style: Cascading U-nets with Multi-level Speaker and Style Modeling for Zero-Shot Voice Cloning

One-shot Voice Conversion For Style Transfer Based On Speaker Adaptation

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding

Self-supervised Context-aware Style Representation for Expressive Speech Synthesis

Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesis

StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech Synthesis

Disentangling Style and Speaker Attributes for TTS Style Transfer

Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation

StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles

AS-Speech: Adaptive Style for Speech Synthesis

StyleSpeech: Parameter-efficient Fine Tuning for Pre-trained Controllable Text-to-Speech

Style-Label-Free: Cross-Speaker Style Transfer by Quantized VAE and Speaker-wise Normalization in Speech Synthesis