Retrieval-augmented generation in multilingual settings

Nadezhda Chirkova,David Rau,Hervé Déjean,Thibault Formal,Stéphane Clinchant,Vassilina Nikoulina

2024-07-02

Abstract:Retrieval-augmented generation (RAG) has recently emerged as a promising solution for incorporating up-to-date or domain-specific knowledge into large language models (LLMs) and improving LLM factuality, but is predominantly studied in English-only settings. In this work, we consider RAG in the multilingual setting (mRAG), i.e. with user queries and the datastore in 13 languages, and investigate which components and with which adjustments are needed to build a well-performing mRAG pipeline, that can be used as a strong baseline in future works. Our findings highlight that despite the availability of high-quality off-the-shelf multilingual retrievers and generators, task-specific prompt engineering is needed to enable generation in user languages. Moreover, current evaluation metrics need adjustments for multilingual setting, to account for variations in spelling named entities. The main limitations to be addressed in future works include frequent code-switching in non-Latin alphabet languages, occasional fluency errors, wrong reading of the provided documents, or irrelevant retrieval. We release the code for the resulting mRAG baseline pipeline at <a class="link-external link-https" href="https://github.com/naver/bergen" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the problem of implementing effective methods for Retrieval-Augmented Generation (RAG) in a multilingual environment. Specifically:

1. **Performance improvement in a multilingual environment**: Investigate whether RAG can bring significant performance improvements in non-English environments as it does in English environments.

2. **Required components and their adjustments**: Explore the components needed to build an efficient multilingual RAG pipeline and the necessary adjustments.

3. **Main limitations of existing components**: Identify the main limitations of current components and propose directions for future work, such as frequent code-switching, occasional fluency errors, misunderstanding of provided documents, or retrieving irrelevant content.

Through these studies, the paper aims to establish a publicly available multilingual RAG benchmark pipeline to promote research on multilingual RAG in zero-shot settings and evaluate the scenario where both user queries and data repositories are in 13 different languages in open-domain question answering tasks.