Abstract:Adapters have emerged as a modular and parameter-efficient approach to (zero-shot) cross-lingual transfer. The established MAD-X framework employs separate language and task adapters which can be arbitrarily combined to perform the transfer of any task to any target language. Subsequently, BAD-X, an extension of the MAD-X framework, achieves improved transfer at the cost of MAD-X's modularity by creating "bilingual" adapters specific to the source-target language pair. In this work, we aim to take the best of both worlds by (i) fine-tuning task adapters adapted to the target language(s) (so-called "target language-ready" (TLR) adapters) to maintain high transfer performance, but (ii) without sacrificing the highly modular design of MAD-X. The main idea of "target language-ready" adapters is to resolve the training-vs-inference discrepancy of MAD-X: the task adapter "sees" the target language adapter for the very first time during inference, and thus might not be fully compatible with it. We address this mismatch by exposing the task adapter to the target language adapter during training, and empirically validate several variants of the idea: in the simplest form, we alternate between using the source and target language adapters during task adapter training, which can be generalized to cycling over any set of language adapters. We evaluate different TLR-based transfer configurations with varying degrees of generality across a suite of standard cross-lingual benchmarks, and find that the most general (and thus most modular) configuration consistently outperforms MAD-X and BAD-X on most tasks and languages.

WAD-X: Improving Zero-shot Cross-lingual Transfer via Adapter-based Word Alignment

Iterative Task-adaptive Pretraining for Unsupervised Word Alignment

Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models

Cross-Lingual Transfer with Target Language-Ready Task Adapters

Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations

Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment

Exploiting Adapters for Cross-Lingual Low-Resource Speech Recognition

AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging

Multi-level multilingual semantic alignment for zero-shot cross-lingual transfer learning

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT

Multiway-Adapter: Adapting Multimodal Large Language Models for Scalable Image-Text Retrieval

Transferable speech-to-text large language model alignment module

Word Alignment by Fine-tuning Embeddings on Parallel Corpora

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment

Alignment Model Adaptation for Domain-Specific Word Alignment.

Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsing

The Impact of Language Adapters in Cross-Lingual Transfer for NLU

Injecting Word Information with Multi-Level Word Adapter for Chinese Spoken Language Understanding