Abstract:The end-to-end speech translation (E2E-ST) model has gradually become a mainstream paradigm due to its low latency and less error propagation. However, it is non-trivial to train such a model well due to the task complexity and data scarcity. The speech-and-text modality differences result in the E2E-ST model performance usually inferior to the corresponding machine translation (MT) model. Based on the above observation, existing methods often use sharingmechanisms to carry out implicit knowledge transfer by imposing various constraints. However, the final model often performs worse on the MT task than the MT model trained alone, which means that the knowledge transfer ability of this method is also limited. To deal with these problems, we propose the FCCL (Fine- and Coarse- Granularity Contrastive Learning) approach for E2E-ST, which makes explicit knowledge transfer through cross-modal multi-grained contrastive learning. A key ingredient of our approach is applying contrastive learning at both sentence- and frame-level to give the comprehensive guide for extracting speech representations containing rich semantic <a class="link-external link-http" href="http://information.In" rel="external noopener nofollow">this http URL</a> addition, we adopt a simple whitening method to alleviate the representation degeneration in the MT model, which adversely affects contrast learning. Experiments on the MuST-C benchmark show that our proposed approach significantly outperforms the state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis indicates that FCCL can free up its capacity from learning grammatical structure information and force more layers to learn semantic information.

Enhancing Multilingual Universal Sentence Embeddings by Monolingual Contrastive Learning

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

Unsupervised Sentence Embedding Model Based on Contrastive Learning

ConCSE: Unified Contrastive Learning and Augmentation for Code-Switched Embeddings

InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

CMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence Embeddings

SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives

Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings

Sparse Contrastive Learning of Sentence Embeddings

HC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding

UNSEE: Unsupervised Non-contrastive Sentence Embeddings

Improving Multi-lingual Alignment Through Soft Contrastive Learning

CLSESSP: Contrastive learning of sentence embedding with strong semantic prototypes

Learning Multilingual Sentence Embeddings From Monolingual Corpus

Stable Contrastive Learning for Self-Supervised Sentence Embeddings With Pseudo-Siamese Mutual Learning

Whitening-based Contrastive Learning of Sentence Embeddings

Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE

Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning

WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings

Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding