Abstract:Non-autoregressive text-to-speech (NAR-TTS) models such as FastSpeech 2 and Glow-TTS can synthesize high-quality speech from the given text in parallel. After analyzing two kinds of generative NAR-TTS models (VAE and normalizing flow), we find that: VAE is good at capturing the long-range semantics features (e.g., prosody) even with small model size but suffers from blurry and unnatural results; and normalizing flow is good at reconstructing the frequency bin-wise details but performs poorly when the number of model parameters is limited. Inspired by these observations, to generate diverse speech with natural details and rich prosody using a lightweight architecture, we propose PortaSpeech, a portable and high-quality generative text-to-speech model. Specifically, 1) to model both the prosody and mel-spectrogram details accurately, we adopt a lightweight VAE with an enhanced prior followed by a flow-based post-net with strong conditional inputs as the main architecture. 2) To further compress the model size and memory footprint, we introduce the grouped parameter sharing mechanism to the affine coupling layers in the post-net. 3) To improve the expressiveness of synthesized speech and reduce the dependency on accurate fine-grained alignment between text and speech, we propose a linguistic encoder with mixture alignment combining hard inter-word alignment and soft intra-word alignment, which explicitly extracts word-level semantic information. Experimental results show that PortaSpeech outperforms other TTS models in both voice quality and prosody modeling in terms of subjective and objective evaluation metrics, and shows only a slight performance degradation when reducing the model parameters to 6.7M (about 4x model size and 3x runtime memory compression ratio compared with FastSpeech 2). Our extensive ablation studies demonstrate that each design in PortaSpeech is effective.

A Synthesis Instance Pruning Approach Based on Virtual Non-Uniform Replacements

ViSPer: A Multilingual TTS Approach Based on VITS Using Deep Feature Loss

On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis

Personalized Lightweight Text-to-Speech: Voice Cloning with Adaptive Structured Pruning

A real-time voice cloning system with multiple algorithms for speech quality improvement

EPIC TTS Models: Empirical Pruning Investigations Characterizing Text-To-Speech Models

Improved unit selection speech synthesis method utilizing subjective evaluation results on synthetic speech

A Synchronized Pruning Composition Algorithm of Weighted Finite State Transducers for Large Vocabulary Speech Recognition

RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion

Accurate and Structured Pruning for Efficient Automatic Speech Recognition

Unit Feature Based Pruning of Large-Scale Speech Corpus Using Decision Tree

PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling

Multi-speaker Prosodic Instance Selection for HMM-based Speech Synthesis

Hierarchical Non-Uniform Unit Selection Based on Prosodic Structure

Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding

PortaSpeech: Portable and High-Quality Generative Text-to-Speech

Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers

QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learning

Applying Syntax$\unicode{x2013}$Prosody Mapping Hypothesis and Prosodic Well-Formedness Constraints to Neural Sequence-to-Sequence Speech Synthesis

High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models

A Hierarchical Viterbi Algorithm For Mandarin Hybrid Speech Synthesis System