Abstract:Large-scale cross-lingual pre-trained language models (xPLMs) have shown effectiveness in cross-lingual sequence labeling tasks (xSL), such as cross-lingual machine reading comprehension (xMRC) by transferring knowledge from a high-resource language to low-resource languages. Despite the great success, we draw an empirical observation that there is a training objective gap between pre-training and fine-tuning stages: e.g., mask language modeling objective requires local understanding of the masked token and the span-extraction objective requires global understanding and reasoning of the input passage/paragraph and question, leading to the discrepancy between pre-training and xMRC. In this paper, we first design a pre-training task tailored for xSL named Cross-lingual Language Informative Span Masking (CLISM) to eliminate the objective gap in a self-supervised manner. Second, we present ContrAstive-Consistency Regularization (CACR), which utilizes contrastive learning to encourage the consistency between representations of input parallel sequences via unsupervised cross-lingual instance-wise training signals during pre-training. By these means, our methods not only bridge the gap between pretrain-finetune, but also enhance PLMs to better capture the alignment between different languages. Extensive experiments prove that our method achieves clearly superior results on multiple xSL benchmarks with limited pre-training data. Our methods also surpass the previous state-of-the-art methods by a large margin in few-shot data settings, where only a few hundred training examples are available.

Cross‐domain Sequence Labelling Using Language Modelling and Parameter Generating

Cross-Lingual Text Image Recognition Via Multi-Task Sequence to Sequence Learning.

Domain Adaptation Meets Zero-Shot Learning: an Annotation-Efficient Approach to Multi-Modality Medical Image Segmentation

Sequence Labeling with Meta-Learning

A Survey on Recent Advances in Sequence Labeling from Deep Learning Models

Label Alignment and Reassignment with Generalist Large Language Model for Enhanced Cross-Domain Named Entity Recognition

Cross-Lingual Adaptation using Structural Correspondence Learning

Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling

Cross-Domain Labeled LDA for Cross-Domain Text Classification

A Joint and Domain-Adaptive Approach to Spoken Language Understanding

Semantic Role Labeling Integrated with Multilevel Linguistic Cues and Bi-LSTM-CRF

Empower Sequence Labeling with Task-Aware Neural Language Model

A New Method for Cross-Lingual-based Semantic Role Labeling

Multi-Task Cross-Lingual Sequence Tagging from Scratch

Cross-Lingual Natural Language Generation via Pre-Training

Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model

Bidirectional Generative Framework for Cross-domain Aspect-based Sentiment Analysis

Multi-Task Domain Adaptation for Language Grounding with 3D Objects

Continual Post-Training of Language Models

Fine-grained Knowledge Fusion for Sequence Labeling Domain Adaptation.

Pseudo-label refinement via hierarchical contrastive learning for source-free unsupervised domain adaptation