Abstract:End-to-end speech translation (ST), which translates source language speech directly into target language text, has garnered significant attention in recent years. Many ST applications require strict length control to ensure that the translation duration matches the length of the source audio, including both speech and pause segments. Previous methods often controlled the number of words or characters generated by the Machine Translation model to approximate the source sentence's length without considering the isochrony of pauses and speech segments, as duration can vary between languages. To address this, we present improvements to the duration alignment component of our sequence-to-sequence ST model. Our method controls translation length by predicting the duration of speech and pauses in conjunction with the translation process. This is achieved by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the translation. The evaluation on the Zh-En test set of CoVoST 2, demonstrates that the proposed Isochrony-Controlled ST achieves 0.92 speech overlap and 8.9 BLEU, which has only a 1.4 BLEU drop compared to the ST baseline.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to achieve isochrony control in language translation from Sino - Tibetan languages to Indo - European languages, so as to ensure that the translated text or speech matches the time length of the source - language audio. Specifically, the paper proposes an improved end - to - end speech translation (ST) model. By predicting the durations of speech and pauses, it controls the translation length, thus maintaining time synchronization with the source audio while generating the translation. This method is especially suitable for application scenarios that require strict length control, such as video dubbing and subtitle synchronization. The paper points out that traditional speech translation methods usually rely on the cascading process of automatic speech recognition (ASR) and machine translation (MT), and this method has limitations in maintaining time synchronization between speech and translated text. To overcome these limitations, the paper proposes a method of directly embedding time information into the decoder to more precisely control the translation length and ensure that the durations of the translated speech and pauses can be aligned with the source audio. Evaluated on the Chinese - to - English translation task, the experimental results show that the proposed Isochrony - Controlled ST method can not only achieve good time synchronization while maintaining high translation quality, but also, under certain configurations, can reach a speech overlap rate of 0.92, with only a 1.4 - point drop in the BLEU score compared to the traditional ST baseline model. This indicates that this method has significant advantages in applications that require strict length control.

Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages

Divergence-Guided Simultaneous Speech Translation

Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding.

Recent Advances in End-to-End Simultaneous Speech Translation

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation

Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation

CoT-ST: Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought

Bridging the Modality Gap for Speech-to-Text Translation

Consecutive Decoding for Speech-to-text Translation

Towards End-to-end Speech-to-text Translation with Two-pass Decoding

Preserving Speaker Information in Direct Speech-to-Speech Translation with Non-Autoregressive Generation and Pretraining

R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translation

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

DiariST: Streaming Speech Translation with Speaker Diarization

Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent

STACL: Simultaneous Translation with Integrated Anticipation and Controllable Latency

Joint Training and Decoding for Multilingual End-to-End Simultaneous Speech Translation

AlloST: Low-resource Speech Translation without Source Transcription

From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation

Pre-Trained Acoustic-and-Textual Modeling for End-To-End Speech-To-Text Translation.

Improving Speech Translation by Understanding the Speech From Latent Code