Abstract:Prosodic boundaries are still crucial to the naturalness of end-to-end speech synthesis systems. This article proposes to use adversarial multi-task learning to predict prosodic boundaries. Adversarial multi-task learning is utilized to transfer knowledge from an auxiliary POS tagging task to a prosodic boundary prediction task. Furthermore, multi-modal embeddings are composed of contextual word and speech embedding features obtained from the pre-trained bidirectional encoder representations from transformers (BERT) model and Speech2Vec. We can utilize linguistic and acoustic information from large amounts of external text and speech data without prosodic boundary labels. At the inference stage, the prosodic boundary predicting model can use the syntactic features learnt from the POS tagging task without any extra computation cost due to only employing the prosodic boundary predicting task to decode. We conducted experiments on Mandarin datasets. The results show that the models using multi-modal embeddings from the pre-trained BERT and Speech2Vec outperform the models trained with single modal embedding. Furthermore, the models trained with adversarial training obtain further performance gains by up to 2.95% in $F_{1}$ score.

Investigating Effect of Rich Syntactic Features on Mandarin Prosodic Boundaries Prediction

Mandarin prosodic word prediction using dependency relationships

Chinese Prosodic Phrasing with Extended Features.

Improving Prosodic Boundaries Prediction For Mandarin Speech Synthesis By Using Enhanced Embedding Feature And Model Fusion Approach

A Study on Chinese Prosodic Hierarchy Prediction Based on Dependency Grammar Analysis

Prosodic Modeling with Rich Syntactic Context in HMM-based Mandarin Speech Synthesis

Acoustic and Linguistic Information Based Chinese Prosodic Boundary Labelling

Dual Audio Encoders Based Mandarin Prosodic Boundary Prediction by Using Multi-Granularity Prosodic Representations

Predicting Syntactic Choice in Mandarin Chinese: a Corpus-Based Analysis of Ba Sentences and SVO Sentences

A Quantitative Study on Information Contribution of Prosody Phrase Boundaries in Chinese Speech

The Mapping Between Syntactic and Prosodic Phrasing in English and Mandarin

Improved Decision Tree Based Method for English Prosodic Phrase Boundary Prediction

A Character-level Span-based Model for Mandarin Prosodic Structure Prediction

Prosodic boundary prediction based on maximum entropy model with error-driven modification

A Maximum Entropy Based Hierarchical Model for Automatic Prosodic Boundary Labeling in Mandarin

Parsing Hierarchical Prosodic Structure For Mandarin Speech Synthesis

Predicting Chinese Prosodic Phrase with Height of Syntax Tree

Adversarial Multi-Task Learning for Mandarin Prosodic Boundary Prediction with Multi-Modal Embeddings

Pitch Prediction for Mandarin TTS with Mutual Prosodic Constraint

Prosodic Word Boundaries Prediction for Mandarin Text-to-Speech

Syntactic Information and Analysis and Prediction of Prosody Structure