Efficiently Identifying Low-Quality Language Subsets in Multilingual Datasets: A Case Study on a Large-Scale Multilingual Audio Dataset

Farhan Samir,Emily P. Ahn,Shreya Prakash,Márton Soskuthy,Vered Shwartz,Jian Zhu

2024-10-06

Abstract:Curating datasets that span multiple languages is challenging. To make the collection more scalable, researchers often incorporate one or more imperfect classifiers in the process, like language identification models. These models, however, are prone to failure, resulting in some language subsets being unreliable for downstream tasks. We introduce a statistical test, the Preference Proportion Test, for identifying such unreliable subsets. By annotating only 20 samples for a language subset, we're able to identify systematic transcription errors for 10 language subsets in a recent large multilingual transcribed audio dataset, X-IPAPack (Zhu et al., 2024). We find that filtering this low-quality data out when training models for the downstream task of phonetic transcription brings substantial benefits, most notably a 25.7% relative improvement on transcribing recordings in out-of-distribution languages. Our method lays a path forward for systematic and reliable multilingual dataset auditing.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to efficiently identify low - quality language subsets in multilingual datasets. Specifically, the authors are concerned with the problem that in large - scale multilingual transcribed audio datasets, some language subsets are unreliable due to the imperfect classifiers (such as language identification models) used in the data collection process. To this end, they introduce a statistical testing method - the Preference Proportion Test (PPT). By only annotating a small number of samples (for example, 20 samples) in each language subset, it can effectively identify systematic transcription errors. This method can not only improve the performance of downstream tasks (such as phoneme transcription), but also provide a path for the systematic and reliable auditing of multilingual datasets. By applying PPT, the authors successfully identified 10 low - quality language subsets in the X - IPAP ACK dataset, and found that after removing these low - quality data, the performance of the model in handling out - of - distribution languages was significantly improved, especially the performance on low - resource languages was significantly improved.