Abstract:Direct speech-to-speech translation (S2ST) with discrete units leverages recent progress in speech representation learning. Specifically, a sequence of discrete representations derived in a self-supervised manner are predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1) Acoustic multimodality: the discrete units derived from speech with same content could be indeterministic due to the acoustic property (e.g., rhythm, pitch, and energy), which causes deterioration of translation accuracy; 2) high latency: current S2ST systems utilize autoregressive models which predict each unit conditioned on the sequence previously generated, failing to take full advantage of parallelism. In this work, we propose TranSpeech, a speech-to-speech translation model with bilateral perturbation. To alleviate the acoustic multimodal problem, we propose bilateral perturbation (BiP), which consists of the style normalization and information enhancement stages, to learn only the linguistic information from speech samples and generate more deterministic representations. With reduced multimodality, we step forward and become the first to establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices and produces high-accuracy results in just a few cycles. Experimental results on three language pairs demonstrate that BiP yields an improvement of 2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, our parallel decoding shows a significant reduction of inference latency, enabling speedup up to 21.4x than autoregressive technique. Audio samples are available at \url{https://TranSpeech.github.io/}

NeurST: Neural Speech Translation Toolkit

ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit

The Volctrans Neural Speech Translation System for IWSLT 2021

Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation

GigaST: A 10,000-hour Pseudo Speech Translation Corpus

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation

AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation

PromptST: Abstract Prompt Learning for End-to-End Speech Translation

End-to-End Speech Translation with Knowledge Distillation

DiariST: Streaming Speech Translation with Speaker Diarization

DUB: Discrete Unit Back-translation for Speech Translation

Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages

Speaker voice normalization for end-to-end speech translation

Non-Parametric Domain Adaptation for End-to-End Speech Translation

The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task

Translatotron 2: High-quality direct speech-to-speech translation with voice preservation

AlloST: Low-resource Speech Translation without Source Transcription

Back Translation for Speech-to-text Translation Without Transcripts

Bridging the Modality Gap for Speech-to-Text Translation

Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding.