Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi

Benjamin Muller,Benoit Sagot,Djamé Seddah

DOI: https://doi.org/10.48550/arXiv.2005.00318

2020-05-01

Abstract:Building natural language processing systems for non standardized and low resource languages is a difficult challenge. The recent success of large-scale multilingual pretrained language models provides new modeling tools to tackle this. In this work, we study the ability of multilingual language models to process an unseen dialect. We take user generated North-African Arabic as our case study, a resource-poor dialectal variety of Arabic with frequent code-mixing with French and written in Arabizi, a non-standardized transliteration of Arabic to Latin script. Focusing on two tasks, part-of-speech tagging and dependency parsing, we show in zero-shot and unsupervised adaptation scenarios that multilingual language models are able to transfer to such an unseen dialect, specifically in two extreme cases: (i) across scripts, using Modern Standard Arabic as a source language, and (ii) from a distantly related language, unseen during pretraining, namely Maltese. Our results constitute the first successful transfer experiments on this dialect, paving thus the way for the development of an NLP ecosystem for resource-scarce, non-standardized and highly variable vernacular languages.

Computation and Language,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is whether multilingual language models can handle unseen dialects. Specifically, taking North African Arabic (Narabizi) as an example, the author studies whether multilingual language models (such as mBERT) can successfully transfer knowledge from known languages to this unseen and resource - poor dialect in zero - sample and unsupervised adaptation scenarios. Narabizi is an Arabic dialect widely used in Algeria. It is often mixed with French and written in the Latin alphabet, but it has no standard spelling or transliteration rules, which makes it very challenging for natural language processing. The paper evaluates the cross - language transfer ability of the model through two tasks - part - of - speech tagging and dependency syntactic analysis, especially the transfer ability from two extreme cases: Modern Standard Arabic (cross - script) and Maltese (a language that is distantly related to North African Arabic and has not been seen in pre - training). The research results pave the way for the development of the natural language processing ecosystem for low - resource, non - standardized and highly variable dialects.