Abstract:Various applications of voice synthesis have been developed independently despite the fact that they generate "voice" as output in common. In addition, the majority of voice synthesis models currently rely on annotated audio data, but it is crucial to scale them to self-supervised datasets in order to effectively capture the wide range of acoustic variations present in human voice, including speaker identity, emotion, and prosody. In this work, we propose Make-A-Voice, a unified framework for synthesizing and manipulating voice signals from discrete representations. Make-A-Voice leverages a "coarse-to-fine" approach to model the human voice, which involves three stages: 1) semantic stage: model high-level transformation between linguistic content and self-supervised semantic tokens, 2) acoustic stage: introduce varying control signals as acoustic conditions for semantic-to-acoustic modeling, and 3) generation stage: synthesize high-fidelity waveforms from acoustic tokens. Make-A-Voice offers notable benefits as a unified voice synthesis framework: 1) Data scalability: the major backbone (i.e., acoustic and generation stage) does not require any annotations, and thus the training data could be scaled up. 2) Controllability and conditioning flexibility: we investigate different conditioning mechanisms and effectively handle three voice synthesis applications, including text-to-speech (TTS), voice conversion (VC), and singing voice synthesis (SVS) by re-synthesizing the discrete voice representations with prompt guidance. Experimental results demonstrate that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models. Audio samples are available at <a class="link-external link-https" href="https://Make-A-Voice.github.io" rel="external noopener nofollow">this https URL</a>

What Does Your Face Sound Like? 3D Face Shape Towards Voice.

Audio-driven Talking Face Video Generation with Natural Head Pose

Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices

Rethinking Voice-Face Correlation: A Geometry View

FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio

RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network

Audio-Driven 3D Facial Animation from In-the-Wild Videos

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

Talking Faces: Audio-to-Video Face Generation

Make-A-Voice: Unified Voice Synthesis With Discrete Representation

Real-time Synthesis of Chinese Visua using MPEG-4 FAP Features in a

Pose-Aware 3D Talking Face Synthesis using Geometry-guided Audio-Vertices Attention

Speech4Mesh: Speech-Assisted Monocular 3D Facial Reconstruction for Speech-Driven 3D Facial Animation

3D Face Synthesis Driven by Personality Impression

Real-time Conversion from a Single 2D Face Image to a 3D Text-Driven Emotive Audio-Visual Avatar

3D Realistic Talking Face Co-Driven by Text and Speech

Meta Talk: Learning To Data-Efficiently Generate Audio-Driven Lip-Synchronized Talking Face With High Definition

Learn2Talk: 3D Talking Face Learns from 2D Talking Face

Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose