Abstract:Modern NLP applications have enjoyed a great boost utilizing neural networks models. Such deep neural models, however, are not applicable to most human languages due to the lack of annotated training data for various NLP tasks. Cross-lingual transfer learning (CLTL) is a viable method for building NLP models for a low-resource target language by leveraging labeled data from other (source) languages. In this work, we focus on the multilingual transfer setting where training data in multiple source languages is leveraged to further boost target language performance. Unlike most existing methods that rely only on language-invariant features for CLTL, our approach coherently utilizes both language-invariant and language-specific features at instance level. Our model leverages adversarial networks to learn language-invariant features, and mixture-of-experts models to dynamically exploit the similarity between the target language and each individual source language. This enables our model to learn effectively what to share between various languages in the multilingual setup. Moreover, when coupled with unsupervised multilingual embeddings, our model can operate in a zero-resource setting where neither target language training data nor cross-lingual resources are available. Our model achieves significant performance gains over prior art, as shown in an extensive set of experiments over multiple text classification and sequence tagging tasks including a large-scale industry dataset.

Cross-Lingual Transfer in Zero-Shot Cross-Language Entity Linking

Zero-Shot Cross-Lingual Transfer in Legal Domain Using Transformer Models

Cross-Lingual Named Entity Recognition Using Parallel Corpus: A New Approach Using XLM-RoBERTa Alignment

What Matters for Neural Cross-Lingual Named Entity Recognition: An Empirical Analysis

Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets

Feature Aggregation in Zero-Shot Cross-Lingual Transfer Using Multilingual BERT.

Cross-Lingual Relevance Transfer for Document Retrieval

Cross-lingual Transfer of Monolingual Models

Neural Cross-Lingual Entity Linking

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study

A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT

Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation.

Zero-Resource Multilingual Model Transfer: Learning What to Share

Improving Zero-Shot Cross-lingual Transfer for Multilingual Question Answering over Knowledge Graph

Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognition

Realistic Zero-Shot Cross-Lingual Transfer in Legal Topic Classification

Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking

Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models

What makes multilingual BERT multilingual?