Abstract:Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.

EfficientTTS 2: Variational End-to-End Text-to-Speech Synthesis and Voice Conversion

ViSPer: A Multilingual TTS Approach Based on VITS Using Deep Feature Loss

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design

VF-Taco2: Towards Fast and Lightweight Synthesis for Autoregressive Models with Variation Autoencoder and Feature Distillation.

Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Data

Efficient Decoding Self-Attention for End-to-end Speech Synthesis

Improving Model Stability and Training Efficiency in Fast, High Quality Expressive Voice Conversion System

Towards Efficiently Learning Monotonic Alignments for Attention-based End-to-End Speech Recognition

FLY-TTS: Fast, Lightweight and High-Quality End-to-End Text-to-Speech Synthesis

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

Efficient Non-Autoregressive GAN Voice Conversion using VQWav2vec Features and Dynamic Convolution

End-to-End Text-to-Speech using Latent Duration based on VQ-VAE

Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder

Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transform

E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS

Optimizing feature fusion for improved zero-shot adaptation in text-to-speech synthesis

AutoTTS: End-to-End Text-to-Speech Synthesis through Differentiable Duration Modeling