Abstract:In recent years, the development of pre-trained models has significantly propelled advancements in natural language processing. However, multilingual sequence-to-sequence pretrained language models (Seq2Seq PLMs) are pretrained on a wide range of languages (e.g., 25 languages), yet often finetuned for specific bilingual tasks (e.g., English–German), leading to domain and task discrepancies between pretraining and finetuning stages, which may lead to sub-optimal downstream performance. In this study, we first illustratively reveal such domain and task discrepancies, and then conduct an in-depth investigation into the side effects that these discrepancies may have on both training dynamic and downstream performance. To alleviate those side effects, we introduce a simple and effective code-switching restoration task (namely code-switching finetuning ) into the standard pretrain-finetune pipeline. Specifically, in the first stage, we recast the downstream data as the self-supervised format used for pretraining, in which the denoising signal is the code-switched cross-lingual phrase. Then, the model is finetuned on downstream task as usual in the second stage. Experiments spanning both natural language generation (12 supervised translations, 30 zero-shot translations, and 2 cross-lingual summarization tasks) and understanding (7 cross-lingual natural language inference tasks) tasks demonstrate that our model consistently and significantly surpasses the standard finetuning strategy. Analyses show that our method introduces negligible computational cost and reduces cross-lingual representation gaps. We have made the code publicly available at: https://github.com/zanchangtong/CSR4mBART .

switch-GLAT: Multilingual Parallel Machine Translation Via Code-Switch Decoder

Glancing Transformer for Non-Autoregressive Neural Machine Translation.

CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units

Code-Switched Text Synthesis in Unseen Language Pairs

Code-Switching Can be Better Aligners: Advancing Cross-Lingual SLU through Representation-Level and Prediction-Level Alignment

Contextual Code Switching for Machine Translation using Language Models

Code-Switching without Switching: Language Agnostic End-to-End Speech Translation

Accelerating Transformer Inference for Translation via Parallel Decoding

Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning

Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation

Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation

The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21

Lossless Speedup of Autoregressive Translation with Generalized Aggressive Decoding

Checks and Strategies for Enabling Code-Switched Machine Translation

Enhancing Code-Switching for Cross-lingual SLU: A Unified View of Semantic and Grammatical Coherence

Transformer-Transducers for Code-Switched Speech Recognition

Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration

TransLLaMa: LLM-based Simultaneous Translation System

Diff-Glat: Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning

Machine-Created Universal Language for Cross-lingual Transfer

Code-switching finetuning: Bridging multilingual pretrained language models for enhanced cross-lingual performance