Abstract:Transformer has demonstrated its great power to learn contextual word representations for multiple languages in a single model. To process multilingual sentences in the model, a learnable vector is usually assigned to each language, which is called "language embedding". The language embedding can be either added to the word embedding or attached at the beginning of the sentence. It serves as a language-specific signal for the Transformer to capture contextual representations across languages. In this paper, we revisit the use of language embedding and identify several problems in the existing formulations. By investigating the interaction between language embedding and word embedding in the self-attention module, we find that the current methods cannot reflect the language-specific word correlation well. Given these findings, we propose a new approach called Cross-lingual Language Projection (XLP) to replace language embedding. For a sentence, XLP projects the word embeddings into language-specific semantic space, and then the projected embeddings will be fed into the Transformer model to process with their language-specific meanings. In such a way, XLP achieves the purpose of appropriately encoding "language" in a multilingual Transformer model. Experimental results show that XLP can freely and significantly boost the model performance on extensive multilingual benchmark datasets. Codes and models will be released at https://github.com/lsj2408/XLP.

Obtaining Better Word Representations via Language Transfer

Improving Word Embeddings via Combining with Complementary Languages.

Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation

Domain structure-based transfer learning for cross-domain word representation

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

Learning Tibetan-Chinese Cross-Lingual Word Embeddings

Revisiting language encoding in learning multilingual representations

Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations

Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?

A Distributed Representation-Based Framework for Cross-Lingual Transfer Parsing.

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models

Improving Automatic Speech Recognition and Speech Translation Via Word Embedding Prediction

Enhancing Isomorphism Between Word Embedding Spaces for Distant Languages Bilingual Lexicon Induction

Cross-lingual Transfer of Sentiment Classifiers

Addressing Domain Adaptation for Chinese Word Segmentation with Instances-Based Transfer Learning.

Improving Vector Space Word Representations Via Kernel Canonical Correlation Analysis

Neural Task Representations as Weak Supervision for Model Agnostic Cross-Lingual Transfer

Unsupervised Cross-Lingual Sentence Representation Learning via Linguistic Isomorphism

A Novel Deep Learning Method for Obtaining Bilingual Corpus from Multilingual Website