Abstract:Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e.g., mel-spectrogram) given a music score. Previous singing acoustic models adopt a simple loss (e.g., L1 and L2) or generative adversarial network (GAN) to reconstruct the acoustic features, while they suffer from over-smoothing and unstable training issues respectively, which hinder the naturalness of synthesized singing. In this work, we propose DiffSinger, an acoustic model for SVS based on the diffusion probabilistic model. DiffSinger is a parameterized Markov chain that iteratively converts the noise into mel-spectrogram conditioned on the music score. By implicitly optimizing variational bound, DiffSinger can be stably trained and generate realistic outputs. To further improve the voice quality and speed up inference, we introduce a shallow diffusion mechanism to make better use of the prior knowledge learned by the simple loss. Specifically, DiffSinger starts generation at a shallow step smaller than the total number of diffusion steps, according to the intersection of the diffusion trajectories of the ground-truth mel-spectrogram and the one predicted by a simple mel-spectrogram decoder. Besides, we propose boundary prediction methods to locate the intersection and determine the shallow step adaptively. The evaluations conducted on a Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work. Extensional experiments also prove the generalization of our methods on text-to-speech task (DiffSpeech). Audio samples: https://diffsinger.github.io. Codes: https://github.com/MoonInTheRiver/DiffSinger.

CSSinger: End-to-End Chunkwise Streaming Singing Voice Synthesis System Based on Conditional Variational Autoencoder

SiFiSinger: A High-Fidelity End-to-End Singing Voice Synthesizer based on Source-filter Model

VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis

VISinger2+: End-to-End Singing Voice Synthesis Augmented by Self-Supervised Learning Representation

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders

Period Singer: Integrating Periodic and Aperiodic Variational Autoencoders for Natural-Sounding End-to-End Singing Voice Synthesis

UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis

Towards Improving the Expressiveness of Singing Voice Synthesis with BERT Derived Semantic Information

TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control

An Empirical Study on End-to-End Singing Voice Synthesis with Encoder-Decoder Architectures

MR-SVS: Singing Voice Synthesis with Multi-Reference Encoder

A Systematic Exploration of Joint-training for Singing Voice Synthesis

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Zero-Shot Singing Voice Synthesis from Musical Score

UniSinger: Unified End-to-End Singing Voice Synthesis with Cross-Modality Information Matching

RealSinger: Ultra-realistic singing voice generation via stochastic differential equations

VITS-based Singing Voice Conversion System with DSPGAN post-processing for SVCC2023

HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models

Singing-Tacotron

Learn to Sing by Listening: Building Controllable Virtual Singer by Unsupervised Learning from Voice Recordings