Abstract:State-of-the-art neural (re)rankers are notoriously data-hungry which -- given the lack of large-scale training data in languages other than English -- makes them rarely used in multilingual and cross-lingual retrieval settings. Current approaches therefore commonly transfer rankers trained on English data to other languages and cross-lingual setups by means of multilingual encoders: they fine-tune all parameters of pretrained massively multilingual Transformers (MMTs, e.g., multilingual BERT) on English relevance judgments, and then deploy them in the target language(s). In this work, we show that two parameter-efficient approaches to cross-lingual transfer, namely Sparse Fine-Tuning Masks (SFTMs) and Adapters, allow for a more lightweight and more effective zero-shot transfer to multilingual and cross-lingual retrieval tasks. We first train language adapters (or SFTMs) via Masked Language Modelling and then train retrieval (i.e., reranking) adapters (SFTMs) on top, while keeping all other parameters fixed. At inference, this modular design allows us to compose the ranker by applying the (re)ranking adapter (or SFTM) trained with source language data together with the language adapter (or SFTM) of a target language. We carry out a large scale evaluation on the CLEF-2003 and HC4 benchmarks and additionally, as another contribution, extend the former with queries in three new languages: Kyrgyz, Uyghur and Turkish. The proposed parameter-efficient methods outperform standard zero-shot transfer with full MMT fine-tuning, while being more modular and reducing training times. The gains are particularly pronounced for low-resource languages, where our approaches also substantially outperform the competitive machine translation-based rankers.

Cross-Lingual Training of Dense Retrievers for Document Retrieval

Cross-Lingual Training with Dense Retrieval for Document Retrieval

Cross-Lingual Relevance Transfer for Document Retrieval

Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

Cross-lingual Information Retrieval with BERT

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-Shot Learning

Adversarial Domain Adaptation for Cross-lingual Information Retrieval with Multilingual BERT

Zero-Shot Cross-Lingual Transfer in Legal Domain Using Transformer Models

Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

Deep Multilabel Multilingual Document Learning for Cross-Lingual Document Retrieval

Cross-Lingual Transfer in Zero-Shot Cross-Language Entity Linking

Learning Cross-Lingual IR from an English Retriever

Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters

Towards Best Practices for Training Multilingual Dense Retrieval Models

Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval

Unsupervised Text Representation Learning via Instruction-Tuning for Zero-Shot Dense Retrieval

Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation

Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

Distillation for Multilingual Information Retrieval

Narrowing the language gap: domain adaptation guided cross-lingual passage re-ranking