Abstract:Current computational approaches for analysing or generating code-mixed sentences do not explicitly model "naturalness" or "acceptability" of code-mixed sentences, but rely on training corpora to reflect distribution of acceptable code-mixed sentences. Modelling human judgement for the acceptability of code-mixed text can help in distinguishing natural code-mixed text and enable quality-controlled generation of code-mixed text. To this end, we construct Cline - a dataset containing human acceptability judgements for English-Hindi (en-hi) code-mixed text. Cline is the largest of its kind with 16,642 sentences, consisting of samples sourced from two sources: synthetically generated code-mixed text and samples collected from online social media. Our analysis establishes that popular code-mixing metrics such as CMI, Number of Switch Points, Burstines, which are used to filter/curate/compare code-mixed corpora have low correlation with human acceptability judgements, underlining the necessity of our dataset. Experiments using Cline demonstrate that simple Multilayer Perceptron (MLP) models trained solely on code-mixing metrics are outperformed by fine-tuned pre-trained Multilingual Large Language Models (MLLMs). Specifically, XLM-Roberta and Bernice outperform IndicBERT across different configurations in challenging data settings. Comparison with ChatGPT's zero and fewshot capabilities shows that MLLMs fine-tuned on larger data outperform ChatGPT, providing scope for improvement in code-mixed tasks. Zero-shot transfer from English-Hindi to English-Telugu acceptability judgments using our model checkpoints proves superior to random baselines, enabling application to other code-mixed language pairs and providing further avenues of research. We publicly release our human-annotated dataset, trained checkpoints, code-mix corpus, and code for data generation and model training.

OffMix-3L: A Novel Code-Mixed Dataset in Bangla-English-Hindi for Offensive Language Identification

SentMix-3L: A Bangla-English-Hindi Code-Mixed Dataset for Sentiment Analysis

BnSentMix: A Diverse Bengali-English Code-Mixed Dataset for Sentiment Analysis

Large scale annotated dataset for code-mix abusive short noisy text

L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models

My Boli: Code-mixed Marathi-English Corpora, Pretrained Language Models and Evaluation Benchmarks

Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi

DravidianCodeMix: sentiment analysis and offensive language identification dataset for Dravidian languages in code-mixed text

Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities

Improving code-mixed hate detection by native sample mixing: A case study for Hindi-English code-mixed scenario

On Importance of Code-Mixed Embeddings for Hate Speech Identification

From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences

Development of POS tagger for English-Bengali Code-Mixed data

Code-mixed Sentiment and Hate-speech Prediction

Analyzing Roles of Classifiers and Code-Mixed factors for Sentiment Identification

Leveraging Language Identification to Enhance Code-Mixed Text Classification

Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation

Preparing Bengali-English Code-Mixed Corpus for Sentiment Analysis of Indian Languages

BharatBhasaNet-A Unified Framework to Identify Indian Code Mix Languages

Deep learning based sentiment analysis and offensive language identification on multilingual code-mixed data

SMPOST: Parts of Speech Tagger for Code-Mixed Indic Social Media Text