mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

Matthieu Futeral,Armel Zebaze,Pedro Ortiz Suarez,Julien Abadji,Rémi Lacroix,Cordelia Schmid,Rachel Bawden,Benoît Sagot

2024-06-13

Abstract:Multimodal Large Language Models (mLLMs) are trained on a large amount of text-image data. While most mLLMs are trained on caption-like data only, Alayrac et al. [2022] showed that additionally training them on interleaved sequences of text and images can lead to the emergence of in-context learning capabilities. However, the dataset they used, M3W, is not public and is only in English. There have been attempts to reproduce their results but the released datasets are English-only. In contrast, current multilingual and multimodal datasets are either composed of caption-like only or medium-scale or fully private data. This limits mLLM research for the 7,000 other languages spoken in the world. We therefore introduce mOSCAR, to the best of our knowledge the first large-scale multilingual and multimodal document corpus crawled from the web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We carefully conduct a set of filtering and evaluation steps to make sure mOSCAR is sufficiently safe, diverse and of good quality. We additionally train two types of multilingual model to prove the benefits of mOSCAR: (1) a model trained on a subset of mOSCAR and captioning data and (2) a model train on captioning data only. The model additionally trained on mOSCAR shows a strong boost in few-shot learning performance across various multilingual image-text tasks and benchmarks, confirming previous findings for English-only mLLMs.

Computation and Language,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper focuses on building a large-scale multilingual and multimodal document-level corpus called mOSCAR for training multimodal large language models (mLLMs). Most existing mLLMs are trained only on caption-style data, but Alayrac et al.'s (2022) research shows that training models on interleaved sequences of text and images can improve the models' contextual learning ability. However, the M3W dataset they used is private and only contains English.

mOSCAR aims to address the following issues:
1. The lack of public, large-scale, multilingual, and multimodal document datasets, which limits mLLM research to over 7,000 other languages in the world.
2. Existing multilingual multimodal datasets are either medium-sized, only contain caption-style data, or completely private.

To address these issues, the researchers crawled data from web pages and created a large-scale mOSCAR corpus consisting of 163 languages, 315 million documents, 214 billion tokens, and 1.2 billion images. They conducted rigorous data filtering and quality assessment to ensure the data's safety, diversity, and high quality. By training two types of multilingual models, they demonstrated that models trained on mOSCAR outperformed in various multilingual image-text tasks and benchmark tests with limited samples.

Furthermore, the paper discusses the comparison with existing datasets, such as English-centric image-text datasets, and the limitations of multilingual image-text datasets, such as translation quality, inadequate conveyance of cultural details, and potential unsafe content.

In conclusion, the creation of mOSCAR aims to promote cross-lingual research on multimodal language models and provide a publicly available, large-scale multilingual multimodal document resource.