Abstract:With recent advances in end-to-end text to speech (TTS), the quality of synthetic data has been significantly improved. Synthesized speech is becoming a feasible alternative to human speech for training speech recognizers. By using multi-speaker TTS, the synthesis architecture manipulations the combination of various speakers, prosodies and speaking styles. It helps to enrich the acoustic diversity and benefits the robustness of automatic speech recognition (ASR). However, the improvement of building acoustic model in ASR system with synthetic data is still limited due to the mismatch between the synthetic and the real data. Human speech is more natural and contains infoxrmation that not exist in synthetic data, such as ambient noise and frequency warping inspired by channel. In this paper, we propose two novel techniques to mitigate the problem: (i) Pre-train TTS model with large dataset, and then transfer it to each speaker for generating synthetic data which is more suitable on the ASR task. (ii) A conditional training method that improves the performance of augmenting real data with synthesized materials. Experimental results show that these methods can significantly improve the building of speech recognition systems using synthetic data. For example, the results on AISHELL-1 dataset show the proposed methods can achieve up to 41.7% relative error reduction in character error rate (CER) compared to the traditional method of building ASR model with synthetic data. Moreover, we observe up to 12.7% relative error reduction by augmenting human speech and synthesized speech with our conditional training method compared to naively using the human speech.

Considering Temporal Connection Between Turns for Conversational Speech Synthesis

Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling

Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis

Target conversation extraction: Source separation using turn-taking dynamics

Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue

Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation

Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis

Real-Time Multimodal Turn-taking Prediction to Enhance Cooperative Dialogue during Human-Agent Interaction

Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding

Multi-view Temporal Alignment for Non-parallel Articulatory-to-Acoustic Speech Synthesis

IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities

Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism

Interactive Conversational Head Generation

High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models

Improving Speech Recognition with Augmented Synthesized Data and Conditional Model Training

Effective Cross-Utterance Language Modeling for Conversational Speech Recognition

Filling the Gap of Utterance-aware and Speaker-aware Representation for Multi-turn Dialogue

Focusing on attention: prosody transfer and adaptative optimization strategy for multi-speaker end-to-end speech synthesis

Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM

Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models