Abstract:Simultaneous speech to speech translation aims to interpret concurrently with the speech in source language, which is of great importance to the real-time understanding of spoken lectures or conversations. Previous methods usually divide this problem into three stages: simultaneous automatic speech recognition (ASR), simultaneous neural machine translation (NMT), and simultaneous text to speech (TTS), which is not end-to-end and suffers from translation delay and error propagation. In this work, we propose SimulS2S, an end-to-end simultaneous speech to speech translation system that directly translates from source-language speech into target-language speech concurrently, which jointly optimizes speech recognition, text translation and speech synthesis in one sequence to sequence model. SimulS2S consists of a speech encoder and a speech decoder both with a speech segmenter and a wait- strategy for simultaneous translation. Since simultaneous speech to speech translation is challenging, we propose several key techniques to help the training of SimulS2S: 1) a curriculum learning mechanism to train the model gradually from full-sentence translation to simultaneous translation; 2) two auxiliary tasks: ASR and S2T (speech to text translation) that share the same encoder with SimulS2S model to help the training of the encoder; 3) knowledge distillation to transfer the knowledge from the cascaded NMT and TTS models to the SimulS2S model. Experiments on Fisher Spanish-English conversation translation datasets demonstrate that SimulS2S 1) achieves low translation delay and reasonable translation quality compared with full …

End-to-End Spoken Language Translation

SimulS2S: End-to-End Simultaneous Speech to Speech Translation

The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation

End-to-End Cross-Lingual Spoken Language Understanding Model with Multilingual Pretraining.

Improved Long-Form Spoken Language Translation with Large Language Models

Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation

End-to-End Automatic Speech Translation of Audiobooks

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding

A Comparative Study on End-to-end Speech to Text Translation

End-to-End Speech-to-Text Translation: A Survey

An Empirical Study of End-To-End Simultaneous Speech Translation Decoding Strategies

Long-Form End-to-End Speech Translation via Latent Alignment Segmentation

Aligning Pre-trained Models for Spoken Language Translation

End-to-End Speech Translation for Code Switched Speech

Large-Scale Unsupervised Pre-Training for End-to-End Spoken Language Understanding.

Towards End-to-end Speech-to-text Translation with Two-pass Decoding

TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data

Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding

A Deep Neural Network Approach To Parallel Sentence Extraction

End-to-End Speech Translation with Knowledge Distillation