Audio-conditioned phonemic and prosodic annotation for building text-to-speech models from unlabeled speech data

Yuma Shirahata,Byeongseon Park,Ryuichi Yamamoto,Kentaro Tachibana

2024-06-12

Abstract:This paper proposes an audio-conditioned phonemic and prosodic annotation model for building text-to-speech (TTS) datasets from unlabeled speech samples. For creating a TTS dataset that consists of label-speech paired data, the proposed annotation model leverages an automatic speech recognition (ASR) model to obtain phonemic and prosodic labels from unlabeled speech samples. By fine-tuning a large-scale pre-trained ASR model, we can construct the annotation model using a limited amount of label-speech paired data within an existing TTS dataset. To alleviate the shortage of label-speech paired data for training the annotation model, we generate pseudo label-speech paired data using text-only corpora and an auxiliary TTS model. This TTS model is also trained with the existing TTS dataset. Experimental results show that the TTS model trained with the dataset created by the proposed annotation method can synthesize speech as naturally as the one trained with a fully-labeled dataset.

Audio and Speech Processing

What problem does this paper attempt to address?

This paper aims to address the problem of building high-quality Text-to-Speech (TTS) systems from unlabeled speech data. Specifically, the authors propose a new annotation model that can directly predict phoneme and prosody labels from unlabeled speech samples. Traditionally, the approach involves using an Automatic Speech Recognition (ASR) model to generate text transcriptions first, followed by text processing to obtain phoneme and prosody information. However, this method struggles to accurately annotate without audio information. To this end, the proposed method directly predicts TTS labels conditioned on the input speech information. To alleviate the issue of insufficient paired labeled speech data required for training the annotation model, the researchers also propose a data augmentation method that uses an auxiliary TTS model to generate pseudo-labeled speech paired data. Experimental results show that the TTS models trained on datasets created using this method can synthesize speech with a naturalness comparable to that of fully labeled datasets. This demonstrates that even with limited labeled data, the method can generate accurate TTS labels and achieve high-quality speech synthesis.

Audio-conditioned phonemic and prosodic annotation for building text-to-speech models from unlabeled speech data

Automatic Prosody Annotation with Pre-Trained Text-Speech Model

TTS-by-TTS: TTS-Driven Data Augmentation for Fast and High-Quality Speech Synthesis

Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis

Almost Unsupervised Text to Speech and Automatic Speech Recognition

Natural language guidance of high-fidelity text-to-speech with synthetic annotations

Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis

Low-Resource Mongolian Speech Synthesis Based on Automatic Prosody Annotation

Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems

Effective Data Augmentation Methods for Neural Text-to-Speech Systems

Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech

Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP

Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation

Dysarthric Speech Recognition Using Pseudo-Labeling, Self-Supervised Feature Learning, and a Joint Multi-Task Learning Approach

Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE

StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations

AudioComposer: Towards Fine-grained Audio Generation with Natural Language Descriptions

Improving Text-To-Audio Models with Synthetic Captions

Adaspeech 2: Adaptive Text to Speech with Untranscribed Data

Prosody Analysis of Audiobooks

Accented Text-to-Speech Synthesis with Limited Data