Abstract:Large language models (LLMs) now exhibit near human-level performance in various tasks, but their performance drops drastically after a handful of high-resource languages due to the imbalance in pre-training data. Inspired by the human process of second language acquisition, particularly code-switching (the practice of language alternation in a conversation), we propose code-switching curriculum learning (CSCL) to enhance cross-lingual transfer for LLMs. CSCL mimics the stages of human language learning by progressively training models with a curriculum consisting of 1) token-level code-switching, 2) sentence-level code-switching, and 3) monolingual corpora. Using Qwen 2 as our underlying model, we demonstrate the efficacy of the CSCL in improving language transfer to Korean, achieving significant performance gains compared to monolingual continual pre-training methods. Ablation studies reveal that both token- and sentence-level code-switching significantly enhance cross-lingual transfer and that curriculum learning amplifies these effects. We also extend our findings into various languages, including Japanese (high-resource) and Indonesian (low-resource), and using two additional models (Gemma 2 and Phi 3.5). We further show that CSCL mitigates spurious correlations between language resources and safety alignment, presenting a robust, efficient framework for more equitable language transfer in LLMs. We observe that CSCL is effective for low-resource settings where high-quality, monolingual corpora for language transfer are hardly available.

CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units

CoVoST 2 and Massively Multilingual Speech-to-Text Translation

CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving

Call Larisa Ivanovna: Code-Switching Fools Multilingual NLU Models

ConCSE: Unified Contrastive Learning and Augmentation for Code-Switched Embeddings

Code-Switched Text Synthesis in Unseen Language Pairs

End-to-End Speech Translation for Code Switched Speech

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition

Transformer-Transducers for Code-Switched Speech Recognition

Simple yet Effective Code-Switching Language Identification with Multitask Pre-Training and Transfer Learning

Towards Real-World Streaming Speech Translation for Code-Switched Speech

switch-GLAT: Multilingual Parallel Machine Translation Via Code-Switch Decoder

A Deep Generative Model for Code-Switched Text

Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages

Code-Switching Curriculum Learning for Multilingual Transfer in LLMs

Can You Traducir This? Machine Translation for Code-Switched Input

Learning Language and Speaker Information for Code-Switch Speech Synthesis with Limited Data.

Unified model for code-switching speech recognition and language identification based on a concatenated tokenizer

Code-Switching without Switching: Language Agnostic End-to-End Speech Translation

Checks and Strategies for Enabling Code-Switched Machine Translation