Abstract:Motivated by the seemingly high accuracy levels of machine learning models in Moldavian versus Romanian dialect identification and the increasing research interest on this topic, we provide a follow-up on the Moldavian versus Romanian Cross-Dialect Topic Identification (MRC) shared task of the VarDial 2019 Evaluation Campaign. The shared task included two sub-task types: one that consisted in discriminating between the Moldavian and Romanian dialects and one that consisted in classifying documents by topic across the two dialects of Romanian. Participants achieved impressive scores, e.g. the top model for Moldavian versus Romanian dialect identification obtained a macro F1 score of 0.895. We conduct a subjective evaluation by human annotators, showing that humans attain much lower accuracy rates compared to machine learning (ML) models. Hence, it remains unclear why the methods proposed by participants attain such high accuracy rates. Our goal is to understand (i) why the proposed methods work so well (by visualizing the discriminative features) and (ii) to what extent these methods can keep their high accuracy levels, e.g. when we shorten the text samples to single sentences or when we use tweets at inference time. A secondary goal of our work is to propose an improved ML model using ensemble learning. Our experiments show that ML models can accurately identify the dialects, even at the sentence level and across different domains (news articles versus tweets). We also analyze the most discriminative features of the best performing models, providing some explanations behind the decisions taken by these models. Interestingly, we learn new dialectal patterns previously unknown to us or to our human annotators. Furthermore, we conduct experiments showing that the machine learning performance on the MRC shared task can be improved through an ensemble based on stacking.

Comparison of various approaches to tagging for the inflectional Slovak language

Comparison of Machine Learning Approaches for Sentiment Analysis in Slovak

Incorporating External POS Tagger for Punctuation Restoration

Slovak morphological tokenizer using the Byte-Pair Encoding algorithm

Evaluating automatic sentence alignment approaches on English-Slovak sentences

Automatic Poetic Metre Detection for Czech Verse

Language-based transfer learning approaches for part-of-speech tagging on Saint Petersburg Corpus of Hagiographic texts (SKAT)

Surfing the modeling of PoS taggers in low-resource scenarios

Feature-Rich Part-of-speech Tagging for Morphologically Complex Languages: Application to Bulgarian

The use of residual analysis to improve the error rate accuracy of machine translation

Open-Source Web Service with Morphological Dictionary-Supplemented Deep Learning for Morphosyntactic Analysis of Czech

Cross-Domain Evaluation of POS Taggers: From Wall Street Journal to Fandom Wiki

Character Eyes: Seeing Language through Character-Level Taggers

Stylistic Fingerprints, POS-tags and Inflected Languages: A Case Study in Polish

Parts-of-Speech Tagger Errors Do Not Necessarily Degrade Accuracy in Extracting Information from Biomedical Text

Evaluating Neural Morphological Taggers for Sanskrit

The Unreasonable Effectiveness of Machine Learning in Moldavian versus Romanian Dialect Identification

Comparative Analysis of Hidden Markov Model and Bidirectional Long Short-Term Memory for POS Tagging in Eastern Armenian

Sentiment Analysis of Czech Texts: An Algorithmic Survey

#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models

Adaptive scheduling for adaptive sampling in POS taggers construction