Abstract:Large-scale cross-lingual pre-trained language models (xPLMs) have shown effectiveness in cross-lingual sequence labeling tasks (xSL), such as cross-lingual machine reading comprehension (xMRC) by transferring knowledge from a high-resource language to low-resource languages. Despite the great success, we draw an empirical observation that there is a training objective gap between pre-training and fine-tuning stages: e.g., mask language modeling objective requires local understanding of the masked token and the span-extraction objective requires global understanding and reasoning of the input passage/paragraph and question, leading to the discrepancy between pre-training and xMRC. In this paper, we first design a pre-training task tailored for xSL named Cross-lingual Language Informative Span Masking (CLISM) to eliminate the objective gap in a self-supervised manner. Second, we present ContrAstive-Consistency Regularization (CACR), which utilizes contrastive learning to encourage the consistency between representations of input parallel sequences via unsupervised cross-lingual instance-wise training signals during pre-training. By these means, our methods not only bridge the gap between pretrain-finetune, but also enhance PLMs to better capture the alignment between different languages. Extensive experiments prove that our method achieves clearly superior results on multiple xSL benchmarks with limited pre-training data. Our methods also surpass the previous state-of-the-art methods by a large margin in few-shot data settings, where only a few hundred training examples are available.

Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models

Breaking the curse of multilinguality with cross-lingual expert language models

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models

Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models

Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer

XLM-E: Cross-lingual Language Model Pre-training Via ELECTRA

Cross-lingual Editing in Multilingual Language Models

Multilingual Large Language Models and Curse of Multilinguality

ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

The Model Arena for Cross-lingual Sentiment Analysis: A Comparative Study in the Era of Large Language Models

Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling

Enhancing Multilingual Speech Generation and Recognition Abilities in LLMs with Constructed Code-switched Data

XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability and Culture Adaptability in LLMs via Mutual Cross-lingual Feed-forward Transplantation

Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance

Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications

Unsupervised cross-lingual representation learning at scale

Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting

On the cross-lingual transferability of monolingual representations

Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners