Abstract:Multilingual sentence encoders are commonly obtained by training multilingual language models to map sentences from different languages into a shared semantic space. As such, they are subject to curse of multilinguality, a loss of monolingual representational accuracy due to parameter sharing. Another limitation of multilingual sentence encoders is the trade-off between monolingual and cross-lingual performance. Training for cross-lingual alignment of sentence embeddings distorts the optimal monolingual structure of semantic spaces of individual languages, harming the utility of sentence embeddings in monolingual tasks. In this work, we address both issues by modular training of sentence encoders, i.e., by separating monolingual specialization from cross-lingual alignment. We first efficiently train language-specific sentence encoders to avoid negative interference between languages (i.e., the curse). We then align all non-English monolingual encoders to the English encoder by training a cross-lingual alignment adapter on top of each, preventing interference with monolingual specialization from the first step. In both steps, we resort to contrastive learning on machine-translated paraphrase data. Monolingual and cross-lingual evaluations on semantic text similarity/relatedness and multiple-choice QA render our modular solution more effective than multilingual sentence encoders, especially benefiting low-resource languages.

Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect

Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect

Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment

SwissBERT: The Multilingual Language Model for Switzerland

Machine Translation of Low-Resource Spoken Dialects: Strategies for Normalizing Swiss German

A Benchmark for Evaluating Machine Translation Metrics on Dialects Without Standard Orthography

SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German

A Swiss German Dictionary: Variation in Speech and Writing

Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer

Dialectal Speech Recognition and Translation of Swiss German Speech to Standard German Text: Microsoft's Submission to SwissText 2021

A Modularized Neural Network with Language-Specific Output Layers for Cross-lingual Voice Conversion

Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders

Spaiche: Extending State-of-the-Art ASR Models to Swiss German Dialects

Dialect Transfer for Swiss German Speech Translation

LLäMmlein: Compact and Competitive German-Only Language Models from Scratch

Text-to-Speech Pipeline for Swiss German -- A comparison

Improving Neural Machine Translation Models with Monolingual Data

Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks

WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

The Impact of Language Adapters in Cross-Lingual Transfer for NLU

Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining