Abstract:Recently, pre-trained language models have achieved remarkable success in a broad range of natural language processing tasks. However, in multilingual setting, it is extremely resource-consuming to pre-train a deep language model over large-scale corpora for each language. Instead of exhaustively pre-training monolingual language models independently, an alternative solution is to pre-train a powerful multilingual deep language model over large-scale corpora in hundreds of languages. However, the vocabulary size for each language in such a model is relatively small, especially for low-resource languages. This limitation inevitably hinders the performance of these multilingual models on tasks such as sequence labeling, wherein in-depth token-level or sentence-level understanding is essential. In this paper, inspired by previous methods designed for monolingual settings, we investigate two approaches (i.e., joint mapping and mixture mapping) based on a pre-trained multilingual model BERT for addressing the out-of-vocabulary (OOV) problem on a variety of tasks, including part-of-speech tagging, named entity recognition, machine translation quality estimation, and machine reading comprehension. Experimental results show that using mixture mapping is more promising. To the best of our knowledge, this is the first work that attempts to address and discuss the OOV issue in multilingual settings.

Improving Bilingual Lexicon Induction on Distant Language Pairs

Machine Translation: 15th China Conference, CCMT 2019, Nanchang, China, September 27–29, 2019, Revised Selected Papers

Bilingual Lexicon Induction from Non-Parallel Data with Minimal Supervision.

How Lexical is Bilingual Lexicon Induction?

Unsupervised Bilingual Lexicon Induction Via Latent Variable Models.

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment

Dual-perspective fusion for word translation enhancement

Bilingual lexicon induction from non-parallel corpora

Bilingual word embedding fusion for robust unsupervised bilingual lexicon induction

Inducing Bilingual Lexica from Non-Parallel Data with Earth Mover's Distance Regularization.

Improving Pre-Trained Multilingual Model with Vocabulary Expansion

A Relaxed Matching Procedure for Unsupervised BLI

Semi-Supervised Bilingual Lexicon Induction with Two-way Interaction

A Simple and Effective Approach to Robust Unsupervised Bilingual Dictionary Induction

Earth Mover's Distance Minimization for Unsupervised Bilingual Lexicon Induction.

Improving Multilingual Models with Language-Clustered Vocabularies

Jointly Learning Bilingual Word Embeddings and Alignments

Semi-Supervised Learning for Bilingual Lexicon Induction

Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport

: Improving Domain-Specific Word Alignment with a General Bilingual Corpus

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems