Improving speech translation by fusing speech and text

Wenbiao Yin,Zhicheng Liu,Chengqi Zhao,Tao Wang,Jian Tong,Rong Ye

2023-05-23

Abstract:In speech translation, leveraging multimodal data to improve model performance and address limitations of individual modalities has shown significant effectiveness. In this paper, we harness the complementary strengths of speech and text, which are disparate modalities. We observe three levels of modality gap between them, denoted by Modal input representation, Modal semantic, and Modal hidden states. To tackle these gaps, we propose \textbf{F}use-\textbf{S}peech-\textbf{T}ext (\textbf{FST}), a cross-modal model which supports three distinct input modalities for translation: speech, text, and fused speech-text. We leverage multiple techniques for cross-modal alignment and conduct a comprehensive analysis to assess its impact on speech translation, machine translation, and fused speech-text translation. We evaluate FST on MuST-C, GigaST, and newstest benchmark. Experiments show that the proposed FST achieves an average 34.0 BLEU on MuST-C En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate that FST does not degrade on MT task, as observed in prior works. Instead, it yields an average improvement of 3.2 BLEU over the pre-trained MT model.

Computation and Language,Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve translation performance by fusing speech and text modalities in speech translation. Specifically, the paper identifies three modal differences between speech and text: modal input representation, modal semantics, and modal hidden states. These differences lead to limitations when directly using a single modality (such as speech only or text only) for translation. For example, the speech signal may contain noise, and the text transcription may contain errors. To overcome these problems, the paper proposes a cross - modal model - Fuse - Speech - Text (FST), which supports three different input modalities: speech, text, and fused speech - text. Through this method, the FST model aims to utilize the complementary advantages of speech and text, thereby achieving better performance in speech translation tasks without degrading the performance of machine translation tasks.

The main contributions of the paper include:
1. Proposing a model that fuses speech and text to improve speech translation, which supports three different input modalities: speech, text, and fused speech - text.
2. Conducting a comprehensive analysis of the modal gap between speech and text and proposing targeted improvement measures to better integrate speech and text.
3. The experimental results show that the model achieves an average BLEU score of 34.0 on the MuST - C dataset (an improvement of 1.1 BLEU scores compared to the existing best method), and also achieves an average improvement of 3.2 BLEU in machine translation tasks.