Abstract:The end-to-end speech translation (E2E-ST) model has gradually become a mainstream paradigm due to its low latency and less error propagation. However, it is non-trivial to train such a model well due to the task complexity and data scarcity. The speech-and-text modality differences result in the E2E-ST model performance usually inferior to the corresponding machine translation (MT) model. Based on the above observation, existing methods often use sharingmechanisms to carry out implicit knowledge transfer by imposing various constraints. However, the final model often performs worse on the MT task than the MT model trained alone, which means that the knowledge transfer ability of this method is also limited. To deal with these problems, we propose the FCCL (Fine- and Coarse- Granularity Contrastive Learning) approach for E2E-ST, which makes explicit knowledge transfer through cross-modal multi-grained contrastive learning. A key ingredient of our approach is applying contrastive learning at both sentence- and frame-level to give the comprehensive guide for extracting speech representations containing rich semantic <a class="link-external link-http" href="http://information.In" rel="external noopener nofollow">this http URL</a> addition, we adopt a simple whitening method to alleviate the representation degeneration in the MT model, which adversely affects contrast learning. Experiments on the MuST-C benchmark show that our proposed approach significantly outperforms the state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis indicates that FCCL can free up its capacity from learning grammatical structure information and force more layers to learn semantic information.

Improving Speech Translation by Understanding the Speech From Latent Code

Learning Semantic Information from Machine Translation to Improve Speech-to-Text Translation

Divergence-Guided Simultaneous Speech Translation

Improving Automatic Speech Recognition and Speech Translation Via Word Embedding Prediction

Bridging the Modality Gap for Speech-to-Text Translation

Towards End-to-end Speech-to-text Translation with Two-pass Decoding

Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation

Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding.

Worse WER, but Better BLEU? Leveraging Word Embedding As Intermediate in Multitask End-to-End Speech Translation

Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation

Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens

CoT-ST: Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought

Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning

Long-Form End-to-End Speech Translation via Latent Alignment Segmentation

R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translation

Investigating Decoder-only Large Language Models for Speech-to-text Translation

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

Pre-training for Speech Translation: CTC Meets Optimal Transport

Latent Space Translation via Semantic Alignment

Multilingual Speech Translation with Efficient Finetuning of Pretrained Models

Towards a Deep Understanding of Multilingual End-to-End Speech Translation