Abstract:Modern NLP applications have enjoyed a great boost utilizing neural networks models. Such deep neural models, however, are not applicable to most human languages due to the lack of annotated training data for various NLP tasks. Cross-lingual transfer learning (CLTL) is a viable method for building NLP models for a low-resource target language by leveraging labeled data from other (source) languages. In this work, we focus on the multilingual transfer setting where training data in multiple source languages is leveraged to further boost target language performance. Unlike most existing methods that rely only on language-invariant features for CLTL, our approach coherently utilizes both language-invariant and language-specific features at instance level. Our model leverages adversarial networks to learn language-invariant features, and mixture-of-experts models to dynamically exploit the similarity between the target language and each individual source language. This enables our model to learn effectively what to share between various languages in the multilingual setup. Moreover, when coupled with unsupervised multilingual embeddings, our model can operate in a zero-resource setting where neither target language training data nor cross-lingual resources are available. Our model achieves significant performance gains over prior art, as shown in an extensive set of experiments over multiple text classification and sequence tagging tasks including a large-scale industry dataset.

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer

Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation.

Zero-Resource Multilingual Model Transfer: Learning What to Share

An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models

Cross-Lingual Training of Dense Retrievers for Document Retrieval

Feature Aggregation in Zero-Shot Cross-Lingual Transfer Using Multilingual BERT.

Can Monolingual Pretrained Models Help Cross-Lingual Classification?

Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappings

Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization

Zero-shot Cross-lingual Transfer is Under-specified Optimization

Zero-Shot Cross-Lingual Transfer in Legal Domain Using Transformer Models

Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models

Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders

Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study

Improving Zero-Shot Translation of Low-Resource Languages

Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selection

Multilingual Speech Translation with Efficient Finetuning of Pretrained Models