Abstract: High-fidelity singing voices usually require higher sampling rate (e.g., 48kHz) to convey expression and emotion. However, higher sampling rate causes the wider frequency band and longer waveform sequences and throws challenges for singing voice synthesis (SVS) in both frequency and time domains. Conventional SVS systems that adopt small sampling rate cannot well address the above challenges. In this paper, we develop HiFiSinger, an SVS system towards high-fidelity singing voice. HiFiSinger consists of a FastSpeech based acoustic model and a Parallel WaveGAN based vocoder to ensure fast training and inference and also high voice quality. To tackle the difficulty of singing modeling caused by high sampling rate (wider frequency band and longer waveform), we introduce multi-scale adversarial training in both the acoustic model and vocoder to improve singing modeling. Specifically, 1) To handle the larger range of frequencies caused by higher sampling rate, we propose a novel sub-frequency GAN (SF-GAN) on mel-spectrogram generation, which splits the full 80-dimensional mel-frequency into multiple sub-bands and models each sub-band with a separate discriminator. 2) To model longer waveform sequences caused by higher sampling rate, we propose a multi-length GAN (ML-GAN) for waveform generation to model different lengths of waveform sequences with separate discriminators. 3) We also introduce several additional designs and findings in HiFiSinger that are crucial for high-fidelity voices, such as adding F0 (pitch) and V/UV (voiced/unvoiced flag) as acoustic features, choosing an appropriate window/hop size for mel-spectrogram, and increasing the receptive field in vocoder for long vowel modeling. Experiment results show that HiFiSinger synthesizes high-fidelity singing voices with much higher quality: 0.32/0.44 MOS gain over 48kHz/24kHz baseline and 0.83 MOS gain over previous SVS systems.

HoloSinger: Semantics and Music Driven Motion Generation with Octahedral Holographic Projection

Text-driven Visual Prosody Generation for Embodied Conversational Agents

RapVerse: Coherent Vocals and Whole-Body Motions Generations from Text

SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model

Generating Holistic 3D Human Motion from Speech

Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communication

Learn to Sing by Listening: Building Controllable Virtual Singer by Unsupervised Learning from Voice Recordings

MusicFace: Music-driven Expressive Singing Face Synthesis

Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions

Towards Variable and Coordinated Holistic Co-Speech Motion Generation

Holographic voice-interactive system with Taylor Rayleigh-Sommerfeld based point cloud gridding

HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis

Real-Time Human Motion Transfer System for Holographic Displays

SingGAN: Generative Adversarial Network for High-Fidelity Singing Voice Generation

Everything2Motion: Synchronizing Diverse Inputs Via a Unified Framework for Human Motion Synthesis

Learning to Generate Diverse Dance Motions with Transformer

RealSinger: Ultra-realistic singing voice generation via stochastic differential equations

UniSinger: Unified End-to-End Singing Voice Synthesis with Cross-Modality Information Matching

Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation

MoMusic: A Motion-Driven Human-AI Collaborative Music Composition and Performing System

Motion Hologram: Jointly optimized hologram generation and motion planning for photorealistic and speckle-free 3D displays via reinforcement learning