Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?

Qingkai Fang,Shaolei Zhang,Zhengrui Ma,Min Zhang,Yang Feng

2024-06-11

Abstract:Recently proposed two-pass direct speech-to-speech translation (S2ST) models decompose the task into speech-to-text translation (S2TT) and text-to-speech (TTS) within an end-to-end model, yielding promising results. However, the training of these models still relies on parallel speech data, which is extremely challenging to collect. In contrast, S2TT and TTS have accumulated a large amount of data and pretrained models, which have not been fully utilized in the development of S2ST models. Inspired by this, in this paper, we first introduce a composite S2ST model named ComSpeech, which can seamlessly integrate any pretrained S2TT and TTS models into a direct S2ST model. Furthermore, to eliminate the reliance on parallel speech data, we propose a novel training method ComSpeech-ZS that solely utilizes S2TT and TTS data. It aligns representations in the latent space through contrastive learning, enabling the speech synthesis capability learned from the TTS data to generalize to S2ST in a zero-shot manner. Experimental results on the CVSS dataset show that when the parallel speech data is available, ComSpeech surpasses previous two-pass models like UnitY and Translatotron 2 in both translation quality and decoding speed. When there is no parallel speech data, ComSpeech-ZS lags behind \name by only 0.7 ASR-BLEU and outperforms the cascaded models.

Computation and Language,Artificial Intelligence,Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve high - quality direct speech - to - speech translation (S2ST) without parallel speech data. Traditional S2ST models rely on parallel speech data for training, which is very difficult to collect, especially for large - scale datasets. In addition, existing S2TT (speech - to - text translation) and TTS (text - to - speech synthesis) models have accumulated a large amount of data and pre - trained models, but these resources have not been fully utilized in the development of S2ST models.

To overcome these challenges, the paper proposes a composite S2ST model - ComSpeech, which can seamlessly integrate any pre - trained S2TT and TTS models. Further, to eliminate the dependence on parallel speech data, the paper also proposes a new training method - ComSpeech - ZS, which only uses S2TT and TTS data and aligns representations in the latent space through contrastive learning, enabling the speech synthesis ability learned from TTS data to be generalized to the S2ST task.

Experimental results show that in the presence of parallel speech data, ComSpeech outperforms previous two - step models (such as UnitY and Translatotron 2) in both translation quality and decoding speed. In the absence of parallel speech data, ComSpeech - ZS is 0.7 ASR - BLEU lower than ComSpeech, but still outperforms the cascaded model.