Abstract:Current computational approaches for analysing or generating code-mixed sentences do not explicitly model "naturalness" or "acceptability" of code-mixed sentences, but rely on training corpora to reflect distribution of acceptable code-mixed sentences. Modelling human judgement for the acceptability of code-mixed text can help in distinguishing natural code-mixed text and enable quality-controlled generation of code-mixed text. To this end, we construct Cline - a dataset containing human acceptability judgements for English-Hindi (en-hi) code-mixed text. Cline is the largest of its kind with 16,642 sentences, consisting of samples sourced from two sources: synthetically generated code-mixed text and samples collected from online social media. Our analysis establishes that popular code-mixing metrics such as CMI, Number of Switch Points, Burstines, which are used to filter/curate/compare code-mixed corpora have low correlation with human acceptability judgements, underlining the necessity of our dataset. Experiments using Cline demonstrate that simple Multilayer Perceptron (MLP) models trained solely on code-mixing metrics are outperformed by fine-tuned pre-trained Multilingual Large Language Models (MLLMs). Specifically, XLM-Roberta and Bernice outperform IndicBERT across different configurations in challenging data settings. Comparison with ChatGPT's zero and fewshot capabilities shows that MLLMs fine-tuned on larger data outperform ChatGPT, providing scope for improvement in code-mixed tasks. Zero-shot transfer from English-Hindi to English-Telugu acceptability judgments using our model checkpoints proves superior to random baselines, enabling application to other code-mixed language pairs and providing further avenues of research. We publicly release our human-annotated dataset, trained checkpoints, code-mix corpus, and code for data generation and model training.

BITS Pilani at HinglishEval: Quality Evaluation for Code-Mixed Hinglish Text Using Transformers

Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text

Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing

Comparative Study of Pre-Trained BERT Models for Code-Mixed Hindi-English Data

HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection

On Importance of Code-Mixed Embeddings for Hate Speech Identification

Leveraging Language Identification to Enhance Code-Mixed Text Classification

Sentiment Analysis of Code-Mixed Social Media Text (Hinglish)

L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models

From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences

Language Identification of Hindi-English tweets using code-mixed BERT

Linguistic Analysis of Hindi-English Mixed Tweets for Depression Detection

Role of Artificial Intelligence in Detection of Hateful Speech for Hinglish Data on Social Media

SMPOST: Parts of Speech Tagger for Code-Mixed Indic Social Media Text

JUNLP@SemEval-2020 Task 9:Sentiment Analysis of Hindi-English code mixed data using Grid Search Cross Validation

Statistical Machine Translation for Indian Languages: Mission Hindi 2

Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches

Marathi To English Neural Machine Translation With Near Perfect Corpus And Transformers

Gui at MixMT 2022 : English-Hinglish: An MT approach for translation of code mixed data

Study of Encoder-Decoder Architectures for Code-Mix Search Query Translation

IIT Gandhinagar at SemEval-2020 Task 9: Code-Mixed Sentiment Classification Using Candidate Sentence Generation and Selection