Abstract:Speech-to-speech translation tasks are commonly tackled by using a three-level cascade system which comprises of speech recognition, machine translation, and speech synthesis. However, this approach suffers from the drawback of error accumulation at each stage. In contrast, the direct speech-to-speech translation model directly converts speech from the source language to the target language without relying on intermediate text generation, thereby avoiding the issue of incorrect transmission in cascading systems. Currently, there exist two categories for direct speech-to-speech translation methods. The first involves mapping the Mel-spectrogram of the source language speech to the Mel-spectrogram of the target language speech. However, this method often encounters challenges in convergence and producing the audible speech for the target language. The second type of methods is to learn a self-supervised discrete representation of the target language using an unlabeled speech corpus. This method entails training a sequence-to-sequence model on a real-world dataset, which then maps the source language speech to the discrete representation of the target language. Finally, a separately trained vocoder is utilized to convert the discrete unit sequence into a speech waveform. Given the limited availability of large-scale Tibetan-Chinese parallel speech corpora, this work adopts the second method to model Tibetan-Chinese speech-to-speech translation tasks. Additionally, a multi-task learning framework is designed in this work to enhance the performance of the speech translation model. Experimental results demonstrate that the Tibetan-Chinese speech-to-speech translation model based on multi-task self-supervised learning outperforms both the model based on spectrogram mapping and the single-task self-supervised learning model in terms of achieving a higher BLUE value.

Transformer Speech Synthesis for Tibetan Lhasa Dialect Based on Multilevel Granular Unit

Neural Speech Synthesis with Transformer Network.

Lhasa-Tibetan Speech Synthesis Using End-to-End Model.

Close to Human Quality TTS with Transformer.

End-to-End Speech Synthesis for Tibetan Multidialect

End-to-End Tibetan-Chinese Speech Translation Based on Multi-task and Multi-level Pre-training

Improving Multi-Speaker Tacotron with Speaker Gating Mechanisms

Bidirectional Decoding Tacotron for Attention Based Neural Speech Synthesis

FastSpeech: Fast, Robust and Controllable Text to Speech

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

Effective Training End-to-End ASR systems for Low-resource Lhasa Dialect of Tibetan Language

DOP-Tacotron: a Fast Chinese TTS System with Local-based Attention

Speech-T: Transducer for Text to Speech and Beyond

Implementation of Tibetan-Chinese Translation Platform Based on LSTM Algorithm

EM-TTS: Efficiently Trained Low-Resource Mongolian Lightweight Text-to-Speech

Hybrid Autoregressive and Non-Autoregressive Transformer Models for Speech Recognition

Using Transfer Learning to Realize Low Resource Dungan Language Speech Synthesis

Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer

Multi-Task Self-Supervised Learning Based Tibetan-Chinese Speech-to-Speech Translation.

A Comparative Study on End-to-End Speech Synthetic Units for Amdo-Tibetan Dialect