Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning

Hao Zhang,Nianwen Si,Yaqi Chen,Wenlin Zhang,Xukui Yang,Dan Qu,Wei-Qiang Zhang
DOI: https://doi.org/10.1109/TASLP.2023.3244521
2023-04-20
Abstract:The end-to-end speech translation (E2E-ST) model has gradually become a mainstream paradigm due to its low latency and less error propagation. However, it is non-trivial to train such a model well due to the task complexity and data scarcity. The speech-and-text modality differences result in the E2E-ST model performance usually inferior to the corresponding machine translation (MT) model. Based on the above observation, existing methods often use sharingmechanisms to carry out implicit knowledge transfer by imposing various constraints. However, the final model often performs worse on the MT task than the MT model trained alone, which means that the knowledge transfer ability of this method is also limited. To deal with these problems, we propose the FCCL (Fine- and Coarse- Granularity Contrastive Learning) approach for E2E-ST, which makes explicit knowledge transfer through cross-modal multi-grained contrastive learning. A key ingredient of our approach is applying contrastive learning at both sentence- and frame-level to give the comprehensive guide for extracting speech representations containing rich semantic <a class="link-external link-http" href="http://information.In" rel="external noopener nofollow">this http URL</a> addition, we adopt a simple whitening method to alleviate the representation degeneration in the MT model, which adversely affects contrast learning. Experiments on the MuST-C benchmark show that our proposed approach significantly outperforms the state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis indicates that FCCL can free up its capacity from learning grammatical structure information and force more layers to learn semantic information.
Computation and Language,Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The main problems that this paper attempts to solve are the modality gap and data scarcity issues encountered during the training process of end - to - end speech translation (E2E - ST) models. Specifically: 1. **Modality gap**: Compared with text, speech is fine - grained and contains more noise, which makes it more difficult to extract semantically rich information from speech. This modality gap results in the performance of E2E - ST models usually being lower than that of the corresponding machine translation (MT) models. 2. **Data scarcity**: E2E - ST is a cross - modality translation task, and its training data is more difficult to collect than that of separate automatic speech recognition (ASR) or machine translation (MT) tasks. Therefore, training a well - performing E2E - ST model is very challenging. To solve these problems, the paper proposes the FCCL (Fine - and Coarse - Granularity Contrastive Learning) method to achieve explicit knowledge transfer from MT to ST through cross - modality multi - granularity contrastive learning. This method provides comprehensive guidance for extracting semantically rich speech representations by applying contrastive learning at the sentence level and the frame level. In addition, a simple whitening method is also adopted to alleviate the representation degradation problem in MT models, thereby improving the effect of contrastive learning. Through these techniques, the paper aims to improve the performance of E2E - ST models, making them significantly superior to existing baseline models on multiple language pairs.