Abstract:Current computational approaches for analysing or generating code-mixed sentences do not explicitly model "naturalness" or "acceptability" of code-mixed sentences, but rely on training corpora to reflect distribution of acceptable code-mixed sentences. Modelling human judgement for the acceptability of code-mixed text can help in distinguishing natural code-mixed text and enable quality-controlled generation of code-mixed text. To this end, we construct Cline - a dataset containing human acceptability judgements for English-Hindi (en-hi) code-mixed text. Cline is the largest of its kind with 16,642 sentences, consisting of samples sourced from two sources: synthetically generated code-mixed text and samples collected from online social media. Our analysis establishes that popular code-mixing metrics such as CMI, Number of Switch Points, Burstines, which are used to filter/curate/compare code-mixed corpora have low correlation with human acceptability judgements, underlining the necessity of our dataset. Experiments using Cline demonstrate that simple Multilayer Perceptron (MLP) models trained solely on code-mixing metrics are outperformed by fine-tuned pre-trained Multilingual Large Language Models (MLLMs). Specifically, XLM-Roberta and Bernice outperform IndicBERT across different configurations in challenging data settings. Comparison with ChatGPT's zero and fewshot capabilities shows that MLLMs fine-tuned on larger data outperform ChatGPT, providing scope for improvement in code-mixed tasks. Zero-shot transfer from English-Hindi to English-Telugu acceptability judgments using our model checkpoints proves superior to random baselines, enabling application to other code-mixed language pairs and providing further avenues of research. We publicly release our human-annotated dataset, trained checkpoints, code-mix corpus, and code for data generation and model training.

Raising the Bar on Acceptability Judgments Classification: An Experiment on ItaCoLA Using ELECTRA

RuCoLA: Russian Corpus of Linguistic Acceptability

Neural Network Acceptability Judgments

Revisiting Acceptability Judgements

JCoLA: Japanese Corpus of Linguistic Acceptability

LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks

Acceptability Judgements via Examining the Topology of Attention Maps

Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmark

Addressing Hallucinations with RAG and NMISS in Italian Healthcare LLM Chatbots

A neural network to identify requests, decisions, and arguments in court rulings on custody

EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts

Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluation

ITALIC: An Italian Intent Classification Dataset

Lessons Learned from EVALITA 2020 and Thirteen Years of Evaluation of Italian Language Technology

From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences

MELA: Multilingual Evaluation of Linguistic Acceptability

Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrations

The Invalsi Benchmarks: measuring Linguistic and Mathematical understanding of Large Language Models in Italian

Camoscio: an Italian Instruction-tuned LLaMA

In-Context Learning for Scalable and Online Hallucination Detection in RAGS

CLAIR-A: Leveraging Large Language Models to Judge Audio Captions