Do "English" Named Entity Recognizers Work Well on Global Englishes?

Alexander Shan,John Bauer,Riley Carlson,Christopher Manning

DOI: https://doi.org/10.18653/v1/2023.findings-emnlp.788

2024-04-21

Abstract:The vast majority of the popular English named entity recognition (NER) datasets contain American or British English data, despite the existence of many global varieties of English. As such, it is unclear whether they generalize for analyzing use of English globally. To test this, we build a newswire dataset, the Worldwide English NER Dataset, to analyze NER model performance on low-resource English variants from around the world. We test widely used NER toolkits and transformer models, including models using the pre-trained contextual models RoBERTa and ELECTRA, on three datasets: a commonly used British English newswire dataset, CoNLL 2003, a more American focused dataset OntoNotes, and our global dataset. All models trained on the CoNLL or OntoNotes datasets experienced significant performance drops-over 10 F1 in some cases-when tested on the Worldwide English dataset. Upon examination of region-specific errors, we observe the greatest performance drops for Oceania and Africa, while Asia and the Middle East had comparatively strong performance. Lastly, we find that a combined model trained on the Worldwide dataset and either CoNLL or OntoNotes lost only 1-2 F1 on both test sets.

Computation and Language,Machine Learning

What problem does this paper attempt to address?

This paper discusses the performance of English Named Entity Recognition (NER) models when dealing with English variants around the world. Most popular NER datasets primarily contain American or British English data, but there are various English variants worldwide. Therefore, the researchers constructed a new dataset - the Global English NER dataset, to analyze the performance of models on English from different regions. They tested popular NER toolkits and models including RoBERTa and ELECTRA, and evaluated them on three datasets: CoNLL 2003 (British English news data), OntoNotes (more focused on American English), and their global dataset.

The study found that when models trained on CoNLL or OntoNotes were applied to the global dataset, the performance significantly decreased, especially with higher error rates in Oceania and Africa, while the performance was relatively better in Asia and the Middle East. By jointly training on the global dataset, the models reduced their loss across all test sets but showed a slight decrease in performance on American and British English texts.

The paper emphasizes that existing NER datasets lack regional diversity, which may result in reduced accuracy of entity recognition by models in a global context. Furthermore, the models may make mistakes when identifying named entities with region-specific meanings, such as mistakenly considering "Japanese Diet" as a medical term rather than the Japanese parliament.

In conclusion, this paper aims to address whether English NER models can effectively handle English variants around the world. It reveals the limitations of existing models in dealing with non-American or British English data and proposes a new dataset to promote wider English representation and enhance model performance.