Abstract:In this paper, we develop DeepSinger, a multi-lingual multi-singer singing voice synthesis (SVS) system, which is built from scratch using singing training data mined from music websites. The pipeline of DeepSinger consists of several steps, including data crawling, singing and accompaniment separation, lyrics-to-singing alignment, data filtration, and singing modeling. Specifically, we design a lyrics-to-singing alignment model to automatically extract the duration of each phoneme in lyrics starting from coarse-grained sentence level to fine-grained phoneme level, and further design a multi-lingual multi-singer singing model based on a feed-forward Transformer to directly generate linear-spectrograms from lyrics, and synthesize voices using Griffin-Lim. DeepSinger has several advantages over previous SVS systems: 1) to the best of our knowledge, it is the first SVS system that directly mines training data from music websites, 2) the lyrics-to-singing alignment model further avoids any human efforts for alignment labeling and greatly reduces labeling cost, 3) the singing model based on a feed-forward Transformer is simple and efficient, by removing the complicated acoustic feature modeling in parametric synthesis and leveraging a reference encoder to capture the timbre of a singer from noisy singing data, and 4) it can synthesize singing voices in multiple languages and multiple singers. We evaluate DeepSinger on our mined singing dataset that consists of about 92 hours data from 89 singers on three languages (Chinese, Cantonese and English). The results demonstrate that with the singing data purely mined from the Web, DeepSinger can synthesize high-quality singing voices in terms of both pitch accuracy and voice naturalness (footnote: Our audio samples are shown in <a class="link-external link-https" href="https://speechresearch.github.io/deepsinger/" rel="external noopener nofollow">this https URL</a>.)

Deep Learning Approaches in Topics of Singing Information Processing

A Deep Learning Based Analysis-Synthesis Framework For Unison Singing

A Survey on Recent Deep Learning-driven Singing Voice Synthesis Systems

More than words: Advancements and challenges in speech recognition for singing

A Deep-Learning Based Framework for Source Separation, Analysis, and Synthesis of Choral Ensembles

Jointly Detecting and Separating Singing Voice: A Multi-Task Approach

DeepSinger: Singing Voice Synthesis with Data Mined From the Web

Research On Singing Voice Detection Based On A Long-Term Recurrent Convolutional Network With Vocal Separation And Temporal Smoothing

Music Deep Learning: Deep Learning Methods for Music Signal Processing—A Review of the State-of-the-Art

An objective evaluation method of vocal singing effect based on artificial intelligence technology

Singing Voice Synthesis Using Deep Autoregressive Neural Networks for Acoustic Modeling

Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models

Deep Learning for Visual Speech Analysis: A Survey

Deep Learning Approach for Singer Voice Classification of Vietnamese Popular Music

Learning Singing From Speech

SingFake: Singing Voice Deepfake Detection

SVDD 2024: The Inaugural Singing Voice Deepfake Detection Challenge

Learn to Sing by Listening: Building Controllable Virtual Singer by Unsupervised Learning from Voice Recordings

Transfer Learning in Vocal Education: Technical Evaluation of Limited Samples Describing Mezzo-soprano

Learn2Sing: Target Speaker Singing Voice Synthesis by learning from a Singing Teacher

SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge Evaluation Plan