Abstract:The end-to-end speech translation (E2E-ST) model has gradually become a mainstream paradigm due to its low latency and less error propagation. However, it is non-trivial to train such a model well due to the task complexity and data scarcity. The speech-and-text modality differences result in the E2E-ST model performance usually inferior to the corresponding machine translation (MT) model. Based on the above observation, existing methods often use sharingmechanisms to carry out implicit knowledge transfer by imposing various constraints. However, the final model often performs worse on the MT task than the MT model trained alone, which means that the knowledge transfer ability of this method is also limited. To deal with these problems, we propose the FCCL (Fine- and Coarse- Granularity Contrastive Learning) approach for E2E-ST, which makes explicit knowledge transfer through cross-modal multi-grained contrastive learning. A key ingredient of our approach is applying contrastive learning at both sentence- and frame-level to give the comprehensive guide for extracting speech representations containing rich semantic <a class="link-external link-http" href="http://information.In" rel="external noopener nofollow">this http URL</a> addition, we adopt a simple whitening method to alleviate the representation degeneration in the MT model, which adversely affects contrast learning. Experiments on the MuST-C benchmark show that our proposed approach significantly outperforms the state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis indicates that FCCL can free up its capacity from learning grammatical structure information and force more layers to learn semantic information.

Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation

Bridging the Modality Gap for Speech-to-Text Translation

Understanding and Bridging the Modality Gap for Speech Translation

Soft Alignment of Modality Space for End-to-end Speech Translation

Tuning Large language model for End-to-end Speech Translation

Pre-Trained Acoustic-and-Textual Modeling for End-To-End Speech-To-Text Translation.

Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning

Speaker Adaptation for End-to-End CTC Models.

Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems

Bridging the Gap Between Pre-Training and Fine-Tuning for End-to-End Speech Translation

An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation

Non-Parametric Domain Adaptation for End-to-End Speech Translation

Discriminative Language Model Adaptation for Mandarin Broadcast Speech Transcription and Translation

Forward-Backward Decoding for Regularizing End-to-End TTS

Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data.

End-to-End Speech Translation with Adversarial Training

Pre-training for Speech Translation: CTC Meets Optimal Transport

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation

Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning

Improving End-to-End Speech Recognition Through Conditional Cross-Modal Knowledge Distillation with Language Model

An Efficient Speaker Adaptation Method for TTS Duration Model