Abstract:Simultaneous speech to speech translation aims to interpret concurrently with the speech in source language, which is of great importance to the real-time understanding of spoken lectures or conversations. Previous methods usually divide this problem into three stages: simultaneous automatic speech recognition (ASR), simultaneous neural machine translation (NMT), and simultaneous text to speech (TTS), which is not end-to-end and suffers from translation delay and error propagation. In this work, we propose SimulS2S, an end-to-end simultaneous speech to speech translation system that directly translates from source-language speech into target-language speech concurrently, which jointly optimizes speech recognition, text translation and speech synthesis in one sequence to sequence model. SimulS2S consists of a speech encoder and a speech decoder both with a speech segmenter and a wait- strategy for simultaneous translation. Since simultaneous speech to speech translation is challenging, we propose several key techniques to help the training of SimulS2S: 1) a curriculum learning mechanism to train the model gradually from full-sentence translation to simultaneous translation; 2) two auxiliary tasks: ASR and S2T (speech to text translation) that share the same encoder with SimulS2S model to help the training of the encoder; 3) knowledge distillation to transfer the knowledge from the cascaded NMT and TTS models to the SimulS2S model. Experiments on Fisher Spanish-English conversation translation datasets demonstrate that SimulS2S 1) achieves low translation delay and reasonable translation quality compared with full …

ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit

SimulS2S: End-to-End Simultaneous Speech to Speech Translation

ESPnet-ST IWSLT 2021 Offline Speech Translation System.

Divergence-Guided Simultaneous Speech Translation

ESPnet2-TTS: Extending the Edge of TTS Research

THE 2020 ESPNET UPDATE: NEW FEATURES, BROADENED APPLICATIONS, PERFORMANCE IMPROVEMENTS, AND FUTURE PLANS

ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding

RECENT DEVELOPMENTS ON ESPNET TOOLKIT BOOSTED BY CONFORMER

Deep Learning Based TTS-STT Model with Transliteration for Indic Languages

NeurST: Neural Speech Translation Toolkit

Espnet-se: end-to-end speech enhancement and separation toolkit designed for asr integration

Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding.

Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models

ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration

CoT-ST: Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought

Towards Real-World Streaming Speech Translation for Code-Switched Speech

Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation

Preserving Speaker Information in Direct Speech-to-Speech Translation with Non-Autoregressive Generation and Pretraining

PolyVoice: Language Models for Speech to Speech Translation

Multilingual Speech-to-Speech Translation into Multiple Target Languages

End-to-End Speech Translation for Code Switched Speech