Concept-to-Speech generation with knowledge sharing for acoustic modelling and utterance filtering

WangXin,LingZhen-Hua,DaiLi-Rong

IF: 3.252

2016-01-01

Computer Speech & Language

Abstract:HighlightsWe present two knowledge sharing approaches for CTS generation.Syntactic features replace prosody phrasing features in HMM-based speech synthesis.Acoustic features are used to filter synt...

What problem does this paper attempt to address?

Concept-to-Speech Generation with Knowledge Sharing for Acoustic Modelling and Utterance Filtering

Xin Wang,Zhen-Hua Ling,Li-Rong Dai

DOI: https://doi.org/10.1016/j.csl.2015.12.003

IF: 3.252

2016-01-01

Computer Speech & Language

Abstract:A Concept-to-Speech (CTS) system converts the conceptual representation of a sentence-to-be-spoken into speech. While some CTS systems consist of independently built text generation and Text-to-Speech (TTS) modules, the majority of the existing CTS systems enhance the connection between these two modules with a prosodic prediction module that utilizes linguistic knowledge from the text generator to predict prosodic features for TTS generation. However, knowledge embodied within the individual modules has the potential to be shared in more ways. This paper describes knowledge sharing for acoustic modelling and utterance filtering in a Mandarin CTS system. First, syntactic information generated by the text generator is propagated to a hidden Markov model (HMM) based acoustic model within the TTS module and replaces the symbolic prosodic phrasing features therein. Our experimental results show that this approach alleviates the local hard-decision problem in automatic prosodic phrasing for Mandarin CTS systems and achieves a comparable performance to the traditional approach without explicit prosodic phrasing. Second, the acoustic features of multiple synthetic utterances expressing the same input concept are utilized to evaluate the utterance candidates. With this ‘post-processing’ mechanism, our CTS system is able to filter out inferior synthetic utterances and find an acceptable candidate to express the input concept.
End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Yuewen Cao,Xixin Wu,Songxiang Liu,Jianwei Yu,Xu Li,Zhiyong Wu,Xunying Liu,Helen Meng

DOI: https://doi.org/10.1109/icassp.2019.8682927

2019-01-01

Abstract:State-of-the-art text-to-speech (TTS) synthesis models can produce monolingual speech with high intelligibility and naturalness. However, when the models are applied to synthesize code-switched (CS) speech, the performance declines seriously. Conventionally, developing a CS TTS system requires multilingual data to incorporate language-specific and cross-lingual knowledge. Recently, end-to-end (E2E) architecture has achieved satisfactory results in monolingual TTS. The architecture enables the training from one end of alphabetic text input to the other end of acoustic feature output. In this paper, we explore the use of E2E framework for CS TTS, using a combination of Mandarin and English monolingual speech corpus uttered by two female speakers. To handle alphabetic input from different languages, we explore two kinds of encoders: (1) shared multilingual encoder with explicit language embedding (LDE); (2) separated monolingual encoder (SPE) for each language. The two systems use identical decoder architecture, where a discriminative code is incorporated to enable the model to generate speech in one speaker's voice consistently. Experiments confirm the effectiveness of the proposed modifications on the E2E TTS framework in terms of quality and speaker similarity of the generated speech. Moreover, our proposed systems can generate controllable foreign-accented speech at character-level using only mixture of monolingual training data.
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding

Chenpeng Du,Yiwei Guo,Feiyu Shen,Zhijun Liu,Zheng Liang,Xie Chen,Shuai Wang,Hui Zhang,Kai Yu

DOI: https://doi.org/10.1609/aaai.v38i16.29747

2024-03-28

Abstract:The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing.

Sound,Audio and Speech Processing
Acoustic Statistical Modeling Based Speech Synthesis Technologies

HU Yu,LING Zhenhua,WANG Renhua,DAI Lirong

DOI: https://doi.org/10.3969/j.issn.1003-0077.2011.06.016

2011-01-01

Abstract:This paper introduces acoustic statistical modeling based speech synthesis technologies.Emphasis is on the research progress contributed by USTC iFLYTEK speech laboratory,which includes: integrate articulatory features and acoustical features for improving the flexibility of acoustical parameters generation;propose a minimum generation error criterion to replace maximum likelihood for improving the synthesized speech quality;use unit selection and waveform concatenation to replace parametric synthesizer and avoid the limitation of speech quality in HMM based parametric synthesis.These innovative techniques improve the performance of speech synthesis systems in naturalness,expressiveness,flexibility and multilingual ability etc.These progresses have made speech synthesis technologies to be widely used in fields of information service of call center,human-machine speech interaction of mobile embedded devices and intelligent speech enabled electronic education systems.
M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis

Jinlong Xue,Yayue Deng,Fengping Wang,Ya Li,Yingming Gao,Jianhua Tao,Jianqing Sun,Jiaen Liang

2023-05-04

Abstract:Conversational text-to-speech (TTS) aims to synthesize speech with proper prosody of reply based on the historical conversation. However, it is still a challenge to comprehensively model the conversation, and a majority of conversational TTS systems only focus on extracting global information and omit local prosody features, which contain important fine-grained information like keywords and emphasis. Moreover, it is insufficient to only consider the textual features, and acoustic features also contain various prosody information. Hence, we propose M2-CTTS, an end-to-end multi-scale multi-modal conversational text-to-speech system, aiming to comprehensively utilize historical conversation and enhance prosodic expression. More specifically, we design a textual context module and an acoustic context module with both coarse-grained and fine-grained modeling. Experimental results demonstrate that our model mixed with fine-grained context information and additionally considering acoustic features achieves better prosody performance and naturalness in CMOS tests.

Sound,Computation and Language,Audio and Speech Processing
Acoustic statistical modeling based new generation speech synthesis technology

WANG Ren-Hua,DAI Li-rong,HU Yu,LING Zhen-hua

2008-01-01

Journal of University of Science and Technology of China

Abstract:This paper introduces acoustic statistical modeling based new generation speech synthesis technology.Emphasis is laid on the research progress in the field of new generation speech synthesis technology contributed by USTC iFlytek speech laboratory,which includes integration articulatory and acoustic features for improving the flexibility of acoustic parameter generation;a minimum generation error(MGE) criterion proposed to replace maximum likelihood for improving synthesized speech quality;use of unit selection and waveform concatenation to replace parametric synthesizer,thus effectively avoiding the limitation of speech quality in HMM based parametric synthesis.These technical innovations may further improve the performance of new generation speech synthesis technology in naturalness,expressiveness,flexibility and multilingual realization,etc.
Cross-Modal Knowledge Distillation Method for Automatic Cued Speech Recognition

Jianrong Wang,Ziyue Tang,Xuewei Li,Mei Yu,Qiang Fang,Li Liu

DOI: https://doi.org/10.48550/arXiv.2106.13686

2021-06-25

Abstract:Cued Speech (CS) is a visual communication system for the deaf or hearing impaired people. It combines lip movements with hand cues to obtain a complete phonetic repertoire. Current deep learning based methods on automatic CS recognition suffer from a common problem, which is the data scarcity. Until now, there are only two public single speaker datasets for French (238 sentences) and British English (97 sentences). In this work, we propose a cross-modal knowledge distillation method with teacher-student structure, which transfers audio speech information to CS to overcome the limited data problem. Firstly, we pretrain a teacher model for CS recognition with a large amount of open source audio speech data, and simultaneously pretrain the feature extractors for lips and hands using CS data. Then, we distill the knowledge from teacher model to the student model with frame-level and sequence-level distillation strategies. Importantly, for frame-level, we exploit multi-task learning to weigh losses automatically, to obtain the balance coefficient. Besides, we establish a five-speaker British English CS dataset for the first time. The proposed method is evaluated on French and British English CS datasets, showing superior CS recognition performance to the state-of-the-art (SOTA) by a large margin.

Multimedia,Sound,Audio and Speech Processing,Image and Video Processing
Prosody Modelling with Pre-trained Cross-utterance Representations for Improved Speech Synthesis

Ya-Jie Zhang,Chao Zhang,Wei Song,Zhengchen Zhang,Youzheng Wu,Xiaodong He

DOI: https://doi.org/10.1109/taslp.2023.3278184

2023-01-01

Abstract:When humans speak multiple utterances in a continuous manner, the prosodic features generated in each utterance are related to those in its neighbouring utterances. Such cross-utterance (CU) dependencies are often ignored by the current neural text-to-speech (TTS) systems, which reduces the naturalness and expressiveness of the synthesized speeches. In this paper, we propose to improve the prosody modelling ability of neural TTS systems using pre-trained CU acoustic and text representations. Such CU acoustic representations are derived using the Wav2Vec 2.0 model (W2V2) from the synthesized audios of the past utterances, while the CU text representations are extracted using the Bidirectional Encoder Representation from Transformers (BERT) model from the scripts of the future utterances. Experimental results on a Mandarin audiobook and an English audiobook showed the naturalness and expressiveness of the synthesized audios were significantly improved by incorporating such pre-trained W2V2 and BERT CU representations into the Fastspeech2 TTS framework.

engineering, electrical & electronic,acoustics
Sub-band Knowledge Distillation Framework for Speech Enhancement

Xiang Hao,Shixue Wen,Xiangdong Su,Yun Liu,Guanglai Gao,Xiaofei Li

DOI: https://doi.org/10.21437/interspeech.2020-1539

2020-01-01

Abstract:In single-channel speech enhancement, methods based on full-band spectral features have been widely studied. However, only a few methods pay attention to non-full-band spectral features. In this paper, we explore a knowledge distillation framework based on sub-band spectral mapping for single-channel speech enhancement. Specifically, we divide the full frequency band into multiple sub-bands and pre-train an elite-level sub-band enhancement model (teacher model) for each sub-band. These teacher models are dedicated to processing their own sub-bands. Next, under the teacher models' guidance, we train a general sub-band enhancement model (student model) that works for all sub-bands. Without increasing the number of model parameters and computational complexity, the student model's performance is further improved. To evaluate our proposed method, we conducted a large number of experiments on an open-source data set. The final experimental results show that the guidance from the elite-level teacher models dramatically improves the student model's performance, which exceeds the full-band model by employing fewer parameters.
A Novel HTS System Using both Continuous HMMs and Discrete HMMs

Jian Yu,Meng Zhang,Jianhua Tao,Xia Wang

2006-01-01

Abstract:The conventional HMM-based speech synthesis system (HTS) has encountered two over-smoothing problems in both time domain and frequency domain. To resolve these two problems, this paper presents a new HTS framework using both continuous HMMs and discrete HMMs. By the replacement of spectral envelope represented by continuous Gaussian distribution with that represented by discrete codevector, the over-smoothing problem in frequency domain can be resolved, and by the replacement of the parameter generation algorithm using dynamic features with a new well-designed codevector selection algorithm, the over-smoothing problem in time domain can also be better resolved. Experimental results show that the using of both continuous HMMs and discrete HMMs significantly improves the voice quality of synthesized speech. Keyword: Speech Synthesis, Hidden Markov Model, Speech Processing.
Improving the Applicability of Knowledge-Enhanced Dialogue Generation Systems by Using Heterogeneous Knowledge from Multiple Sources

Sixing Wu,Minghui Wang,Ying Li,Dawei Zhang,Zhonghai Wu

DOI: https://doi.org/10.1145/3488560.3498393

2022-01-01

Abstract:Traditional conversational systems can only access the given query during the response generation, leading to meaningless responses. To this end, researchers proposed to enhance dialogue generation by integrating external knowledge. Although such methods have achieved remarkable gains, the use of only single-source knowledge often makes existing knowledge-enhanced methods degenerate into traditional models in real scenarios because of the insufficient knowledge coverage of single-source knowledge. To improve the applicability of knowledge-enhanced methods, we propose two novel frameworks to use heterogeneous knowledge from multiple sources. We first propose an MHKD-Seq2Seq framework, which can use different heterogeneous knowledge by identifying abstractlevel knowledge behaviors; meanwhile, a Diffuse-Aggregate scheme is used to process multiple knowledge simultaneously and produce a unified result. The next framework MHKD-ARPLM can leverage the advantages of pretrained language models with Knowledge Linearization techniques. In experiments, we collected dialogues from previously open-released datasets and built a multisource knowledge-aligned dataset TriKE-Weibo, which involves three knowledge sources: commonsense, texts, and infobox tables. Extensive evaluations demonstrate the performance leadership of our approaches against competitive baseline models.
Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data

Liang-Hsuan Tseng,Zih-Ching Chen,Wei-Shun Chang,Cheng-Kuang Lee,Tsung-Ren Huang,Hung-yi Lee

2024-07-15

Abstract:Recent advances in automatic speech recognition (ASR) often rely on large speech foundation models for generating high-quality transcriptions. However, these models can be impractical due to limited computing resources. The situation is even more severe in terms of more realistic or difficult scenarios, such as code-switching ASR (CS-ASR). To address this, we present a framework for developing more efficient models for CS-ASR through knowledge distillation using realistic speech-only data. Our proposed method, Leave No Knowledge Behind During Knowledge Distillation (K$^2$D), leverages both the teacher model's knowledge and additional insights from a small auxiliary model. We evaluate our approach on two in-domain and two out-domain datasets, demonstrating that K$^2$D is effective. By conducting K$^2$D on the unlabeled realistic data, we have successfully obtained a 2-time smaller model with 5-time faster generation speed while outperforming the baseline methods and the teacher model on all the testing sets. We have made our model publicly available on Hugging Face (<a class="link-external link-https" href="https://huggingface.co/andybi7676/k2d-whisper.zh-en" rel="external noopener nofollow">this https URL</a>).

Audio and Speech Processing,Computation and Language,Sound
Learn2Sing 2.0: Diffusion and Mutual Information-Based Target Speaker SVS by Learning from Singing Teacher

Heyang Xue,Xinsheng Wang,Yongmao Zhang,Lei Xie,Pengcheng Zhu,Mengxiao Bi

DOI: https://doi.org/10.48550/arXiv.2203.16408

2022-05-26

Abstract:Building a high-quality singing corpus for a person who is not good at singing is non-trivial, thus making it challenging to create a singing voice synthesizer for this person. Learn2Sing is dedicated to synthesizing the singing voice of a speaker without his or her singing data by learning from data recorded by others, i.e., the singing teacher. Inspired by the fact that pitch is the key style factor to distinguish singing from speaking voice, the proposed Learn2Sing 2.0 first generates the preliminary acoustic feature with averaged pitch value in the phone level, which allows the training of this process for different styles, i.e., speaking or singing, share same conditions except for the speaker information. Then, conditioned on the specific style, a diffusion decoder, which is accelerated by a fast sampling algorithm during the inference stage, is adopted to gradually restore the final acoustic feature. During the training, to avoid the information confusion of the speaker embedding and the style embedding, mutual information is employed to restrain the learning of speaker embedding and style embedding. Experiments show that the proposed approach is capable of synthesizing high-quality singing voice for the target speaker without singing data with 10 decoding steps.

Sound,Audio and Speech Processing
FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis

Yifan Hu,Rui Liu,Guanglai Gao,Haizhou Li

DOI: https://doi.org/10.48550/arXiv.2210.15360

2022-10-27

Abstract:Conversational Text-to-Speech (TTS) aims to synthesis an utterance with the right linguistic and affective prosody in a conversational context. The correlation between the current utterance and the dialogue history at the utterance level was used to improve the expressiveness of synthesized speech. However, the fine-grained information in the dialogue history at the word level also has an important impact on the prosodic expression of an utterance, which has not been well studied in the prior work. Therefore, we propose a novel expressive conversational TTS model, termed as FCTalker, that learn the fine and coarse grained context dependency at the same time during speech generation. Specifically, the FCTalker includes fine and coarse grained encoders to exploit the word and utterance-level context dependency. To model the word-level dependencies between an utterance and its dialogue history, the fine-grained dialogue encoder is built on top of a dialogue BERT model. The experimental results show that the proposed method outperforms all baselines and generates more expressive speech that is contextually appropriate. We release the source code at: <a class="link-external link-https" href="https://github.com/walker-hyf/FCTalker" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Sound,Audio and Speech Processing
End-to-End Speech Translation with Knowledge Distillation

Yuchen Liu,Hao Xiong,Zhongjun He,Jiajun Zhang,Hua Wu,Haifeng Wang,Chengqing Zong

DOI: https://doi.org/10.48550/arXiv.1904.08075

2019-04-17

Abstract:End-to-end speech translation (ST), which directly translates from source language speech into target language text, has attracted intensive attentions in recent years. Compared to conventional pipeline systems, end-to-end ST models have advantages of lower latency, smaller model size and less error propagation. However, the combination of speech recognition and text translation in one model is more difficult than each of these two tasks. In this paper, we propose a knowledge distillation approach to improve ST model by transferring the knowledge from text translation model. Specifically, we first train a text translation model, regarded as a teacher model, and then ST model is trained to learn output probabilities from teacher model through knowledge distillation. Experiments on English- French Augmented LibriSpeech and English-Chinese TED corpus show that end-to-end ST is possible to implement on both similar and dissimilar language pairs. In addition, with the instruction of teacher model, end-to-end ST model can gain significant improvements by over 3.5 BLEU points.

Computation and Language
UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis

Yi Lei,Shan Yang,Xinsheng Wang,Qicong Xie,Jixun Yao,Lei Xie,Dan Su

DOI: https://doi.org/10.48550/arXiv.2212.01546

2022-12-06

Abstract:Text-to-speech (TTS) and singing voice synthesis (SVS) aim at generating high-quality speaking and singing voice according to textual input and music scores, respectively. Unifying TTS and SVS into a single system is crucial to the applications requiring both of them. Existing methods usually suffer from some limitations, which rely on either both singing and speaking data from the same person or cascaded models of multiple tasks. To address these problems, a simplified elegant framework for TTS and SVS, named UniSyn, is proposed in this paper. It is an end-to-end unified model that can make a voice speak and sing with only singing or speaking data from this person. To be specific, a multi-conditional variational autoencoder (MC-VAE), which constructs two independent latent sub-spaces with the speaker- and style-related (i.e. speak or sing) conditions for flexible control, is proposed in UniSyn. Moreover, supervised guided-VAE and timbre perturbation with the Wasserstein distance constraint are leveraged to further disentangle the speaker timbre and style. Experiments conducted on two speakers and two singers demonstrate that UniSyn can generate natural speaking and singing voice without corresponding training data. The proposed approach outperforms the state-of-the-art end-to-end voice generation work, which proves the effectiveness and advantages of UniSyn.

Sound,Audio and Speech Processing
Msdtron: a high-capability multi-speaker speech synthesis system for diverse data using characteristic information

Qinghua Wu,Quanbo Shen,Jian Luan,YuJun Wang

DOI: https://doi.org/10.48550/arXiv.2107.03065

2022-02-11

Abstract:In multi-speaker speech synthesis, data from a number of speakers usually tend to have great diversity due to the fact that the speakers may differ largely in ages, speaking styles, emotions, and so on. It is important but challenging to improve the modeling capabilities for multi-speaker speech synthesis. To address the issue, this paper proposes a high-capability speech synthesis system, called Msdtron, in which 1) a representation of the harmonic structure of speech, called excitation spectrogram, is designed to directly guide the learning of harmonics in mel-spectrogram. 2) conditional gated LSTM (CGLSTM) is proposed to control the flow of text content information through the network by re-weighting the gates of LSTM using speaker information. The experiments show a significant reduction in reconstruction error of mel-spectrogram in the training of the multi-speaker model, and a great improvement is observed in the subjective evaluation of speaker adapted model.

Sound,Audio and Speech Processing
A context-aware knowledge transferring strategy for CTC-based ASR

Ke-Han Lu,Kuan-Yu Chen

DOI: https://doi.org/10.48550/arXiv.2210.06244

2022-10-12

Abstract:Non-autoregressive automatic speech recognition (ASR) modeling has received increasing attention recently because of its fast decoding speed and superior performance. Among representatives, methods based on the connectionist temporal classification (CTC) are still a dominating stream. However, the theoretically inherent flaw, the assumption of independence between tokens, creates a performance barrier for the school of works. To mitigate the challenge, we propose a context-aware knowledge transferring strategy, consisting of a knowledge transferring module and a context-aware training strategy, for CTC-based ASR. The former is designed to distill linguistic information from a pre-trained language model, and the latter is framed to modulate the limitations caused by the conditional independence assumption. As a result, a knowledge-injected context-aware CTC-based ASR built upon the wav2vec2.0 is presented in this paper. A series of experiments on the AISHELL-1 and AISHELL-2 datasets demonstrate the effectiveness of the proposed method.

Computation and Language,Sound,Audio and Speech Processing
Single-stage TTS with Masked Audio Token Modeling and Semantic Knowledge Distillation

Gerard I. Gállego,Roy Fejgin,Chunghsin Yeh,Xiaoyu Liu,Gautam Bhattacharya

2024-09-17

Abstract:Audio token modeling has become a powerful framework for speech synthesis, with two-stage approaches employing semantic tokens remaining prevalent. In this paper, we aim to simplify this process by introducing a semantic knowledge distillation method that enables high-quality speech generation in a single stage. Our proposed model improves speech quality, intelligibility, and speaker similarity compared to a single-stage baseline. Although two-stage systems still lead in intelligibility, our model significantly narrows the gap while delivering comparable speech quality. These findings showcase the potential of single-stage models to achieve efficient, high-quality TTS with a more compact and streamlined architecture.

Sound,Artificial Intelligence,Audio and Speech Processing,Signal Processing
DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speech

Sen Liu,Yiwei Guo,Chenpeng Du,Xie Chen,Kai Yu

2023-06-25

Abstract:Although high-fidelity speech can be obtained for intralingual speech synthesis, cross-lingual text-to-speech (CTTS) is still far from satisfactory as it is difficult to accurately retain the speaker timbres(i.e. speaker similarity) and eliminate the accents from their first language(i.e. nativeness). In this paper, we demonstrated that vector-quantized(VQ) acoustic feature contains less speaker information than mel-spectrogram. Based on this finding, we propose a novel dual speaker embedding TTS (DSE-TTS) framework for CTTS with authentic speaking style. Here, one embedding is fed to the acoustic model to learn the linguistic speaking style, while the other one is integrated into the vocoder to mimic the target speaker's timbre. Experiments show that by combining both embeddings, DSE-TTS significantly outperforms the state-of-the-art SANE-TTS in cross-lingual synthesis, especially in terms of nativeness.

Sound,Computation and Language,Audio and Speech Processing

Concept-to-Speech generation with knowledge sharing for acoustic modelling and utterance filtering

Concept-to-Speech Generation with Knowledge Sharing for Acoustic Modelling and Utterance Filtering

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding

Acoustic Statistical Modeling Based Speech Synthesis Technologies

M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis

Acoustic statistical modeling based new generation speech synthesis technology

Cross-Modal Knowledge Distillation Method for Automatic Cued Speech Recognition

Prosody Modelling with Pre-trained Cross-utterance Representations for Improved Speech Synthesis

Sub-band Knowledge Distillation Framework for Speech Enhancement

A Novel HTS System Using both Continuous HMMs and Discrete HMMs

Improving the Applicability of Knowledge-Enhanced Dialogue Generation Systems by Using Heterogeneous Knowledge from Multiple Sources

Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data

Learn2Sing 2.0: Diffusion and Mutual Information-Based Target Speaker SVS by Learning from Singing Teacher

FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis

End-to-End Speech Translation with Knowledge Distillation

UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis

Msdtron: a high-capability multi-speaker speech synthesis system for diverse data using characteristic information

A context-aware knowledge transferring strategy for CTC-based ASR

Single-stage TTS with Masked Audio Token Modeling and Semantic Knowledge Distillation

DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speech