Abstract:Speech-to-speech translation tasks are commonly tackled by using a three-level cascade system which comprises of speech recognition, machine translation, and speech synthesis. However, this approach suffers from the drawback of error accumulation at each stage. In contrast, the direct speech-to-speech translation model directly converts speech from the source language to the target language without relying on intermediate text generation, thereby avoiding the issue of incorrect transmission in cascading systems. Currently, there exist two categories for direct speech-to-speech translation methods. The first involves mapping the Mel-spectrogram of the source language speech to the Mel-spectrogram of the target language speech. However, this method often encounters challenges in convergence and producing the audible speech for the target language. The second type of methods is to learn a self-supervised discrete representation of the target language using an unlabeled speech corpus. This method entails training a sequence-to-sequence model on a real-world dataset, which then maps the source language speech to the discrete representation of the target language. Finally, a separately trained vocoder is utilized to convert the discrete unit sequence into a speech waveform. Given the limited availability of large-scale Tibetan-Chinese parallel speech corpora, this work adopts the second method to model Tibetan-Chinese speech-to-speech translation tasks. Additionally, a multi-task learning framework is designed in this work to enhance the performance of the speech translation model. Experimental results demonstrate that the Tibetan-Chinese speech-to-speech translation model based on multi-task self-supervised learning outperforms both the model based on spectrogram mapping and the single-task self-supervised learning model in terms of achieving a higher BLUE value.

Tibetan Language Speech Recognition Model Based on Active Learning and Semi-Supervised Learning

Active Learning and Semi-Supervised Learning in Tibetan Language Speech Recognition

Tibetan Language Continuous Speech Recognition Based On Active Ws-Dbn

Finer-grained Modeling Units-Based Meta-Learning for Low-resource Tibetan Speech Recognition

Selection of acoustic modeling unit for Tibetan speech recognition based on deep learning

Cross-Language Transfer Learning-based Lhasa-Tibetan Speech Recognition

Effective Fine-tuning Method for Tibetan Low-resource Dialect Speech Recognition

Multi-Task Self-Supervised Learning Based Tibetan-Chinese Speech-to-Speech Translation.

Grammar-Based Semi-Supervised Incremental Learning in Automatic Speech Recognition and Labeling

Effective Training End-to-End ASR systems for Low-resource Lhasa Dialect of Tibetan Language

Exploring Task-Diverse Meta-Learning on Tibetan Multi-Dialect Speech Recognition

Tibetan Multi-dialect Speech and Dialect Identity Recognition

Tibetan Syllable Prediction with Pre-trained Cross-lingual Language Model

End-to-End-Based Tibetan Multitask Speech Recognition.

Tibetan-Mandarin Bilingual Speech Recognition Based on End-to-end Framework

Tibetan Multi-Dialect Speech Recognition Using Latent Regression Bayesian Network and End-To-End Mode

Phone Recognition for Lhasa-Tibetan Based on Articulatory Features Augmentation Learning

Tibetan Language Continuous Speech Recognition Based on Dynamic Bayesian Network

Unsupervised Active Learning: Optimizing Labeling Cost-Effectiveness for Automatic Speech Recognition

Fast Construction of Speech Recognition Model Based on Sample Selection Strategy

Towards Language-universal Mandarin-English Speech Recognition with Unsupervised Label Synchronous Adaptation