Abstract:The gap between speech and text modalities is a major challenge in speech-to-text translation (ST). Different methods have been proposed to reduce this gap, but most of them require architectural changes in ST training. In this work, we propose to mitigate this issue at the pre-training stage, requiring no change in the ST model. First, we show that the connectionist temporal classification (CTC) loss can reduce the modality gap by design. We provide a quantitative comparison with the more common cross-entropy loss, showing that pre-training with CTC consistently achieves better final ST accuracy. Nevertheless, CTC is only a partial solution and thus, in our second contribution, we propose a novel pre-training method combining CTC and optimal transport to further reduce this gap. Our method pre-trains a Siamese-like model composed of two encoders, one for acoustic inputs and the other for textual inputs, such that they produce representations that are close to each other in the Wasserstein space. Extensive experiments on the standard CoVoST-2 and MuST-C datasets show that our pre-training method applied to the vanilla encoder-decoder Transformer achieves state-of-the-art performance under the no-external-data setting, and performs on par with recent strong multi-task learning systems trained with external data. Finally, our method can also be applied on top of these multi-task systems, leading to further improvements for these models. Code and pre-trained models are available at <a class="link-external link-https" href="https://github.com/formiel/fairseq" rel="external noopener nofollow">this https URL</a>.

CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training

On joint training with interfaces for spoken language understanding

Using Bidirectional Transformer-CRF for Spoken Language Understanding.

Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation

SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training

Bridging Speech and Textual Pre-trained Models with Unsupervised ASR.

A Streaming End-to-End Framework for Spoken Language Understanding.

Cross-Modal Semantic Alignment before Fusion for Two-Pass End-to-End Spoken Language Understanding

CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition

SLM: Bridge the thin gap between speech and text foundation models

Decoupling Breaks Data Barriers: A Decoupled Pre-training Framework for Multi-intent Spoken Language Understanding

End-to-End Cross-Lingual Spoken Language Understanding Model with Multilingual Pretraining.

SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data

Pre-training for Speech Translation: CTC Meets Optimal Transport

End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoders

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding

Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systems

Understanding Semantics from Speech Through Pre-training

Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens

Boosting Continuous Sign Language Recognition via Cross Modality Augmentation