Abstract:The end-to-end speech translation (E2E-ST) model has gradually become a mainstream paradigm due to its low latency and less error propagation. However, it is non-trivial to train such a model well due to the task complexity and data scarcity. The speech-and-text modality differences result in the E2E-ST model performance usually inferior to the corresponding machine translation (MT) model. Based on the above observation, existing methods often use sharingmechanisms to carry out implicit knowledge transfer by imposing various constraints. However, the final model often performs worse on the MT task than the MT model trained alone, which means that the knowledge transfer ability of this method is also limited. To deal with these problems, we propose the FCCL (Fine- and Coarse- Granularity Contrastive Learning) approach for E2E-ST, which makes explicit knowledge transfer through cross-modal multi-grained contrastive learning. A key ingredient of our approach is applying contrastive learning at both sentence- and frame-level to give the comprehensive guide for extracting speech representations containing rich semantic <a class="link-external link-http" href="http://information.In" rel="external noopener nofollow">this http URL</a> addition, we adopt a simple whitening method to alleviate the representation degeneration in the MT model, which adversely affects contrast learning. Experiments on the MuST-C benchmark show that our proposed approach significantly outperforms the state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis indicates that FCCL can free up its capacity from learning grammatical structure information and force more layers to learn semantic information.

FC-MTLF: A Fine- and Coarse-grained Multi-Task Learning Framework for Cross-Lingual Spoken Language Understanding.

Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning

MCLF: A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding

Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding

GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding

DiffSLU: Knowledge Distillation Based Diffusion Model for Cross-Lingual Spoken Language Understanding

Joint Spoken Language Understanding And Domain Adaptive Language Modeling

End-to-End Cross-Lingual Spoken Language Understanding Model with Multilingual Pretraining.

Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning

Multi-Domain Spoken Language Understanding Using Domain- and Task-Aware Parameterization

Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages

ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding

A Joint and Domain-Adaptive Approach to Spoken Language Understanding

Enhancing Multilingual Speech Generation and Recognition Abilities in LLMs with Constructed Code-switched Data

Multi-Task Learning for Multiple Language Translation.

Competence-based Curriculum Learning for Multilingual Machine Translation

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

A multitask co-training framework for improving speech translation by leveraging speech recognition and machine translation tasks

I^2KD-SLU: an Intra-Inter Knowledge Distillation Framework for Zero-Shot Cross-Lingual Spoken Language Understanding

I$^2$KD-SLU: An Intra-Inter Knowledge Distillation Framework for Zero-Shot Cross-Lingual Spoken Language Understanding

Semi-Supervised Training Using Adversarial Multi-Task Learning for Spoken Language Understanding.