Abstract:Natural language inference (NLI), also known as textual entailment recognition (TER), is a crucial task in natural language processing that combines many fundamental aspects of language understanding. Despite the recent significant advancement in NLI, primarily driven by the development of diverse large-scale datasets, most of the progress has been confined to English. This is attributed to the scarcity of human-annotated corpora for most other languages, notably Arabic. In this paper, we present an Arabic NLI dataset called ArEntail , consisting of 6000 sentence pairs collected from news headlines and manually labeled to indicate whether an entailment relationship links the sentences or not without resorting to machine translation from English datasets. To our knowledge, this is the largest yet human-crafted NLI dataset for the Arabic language. We offer various data analyses and establish baseline results using state-of-the-art pre-trained models for Arabic, in addition to a human-based evaluation. Our findings revealed that AraBERT-base v2, the best-performing model, achieves an accuracy of 93%, revealing a gap of 2.6% compared to human performance and presenting a valuable opportunity for further advancements in modeling techniques in future research. Besides, the "hypothesis-only" baseline performance baseline closely resembles a random guesser's, indicating the rarity of annotation artifacts compared to prior NLI English benchmarks. We also evaluated GPT-3.5-turbo in zero-shot and few-shot Arabic NLI learning scenarios and observed promising outcomes with a cautious approach, awaiting strong clues for predicting the presence of the entailment relationship.

SOME EFFECTS OF A PARTIALLY PURIFIED LYMPHOCYTE‐INHIBITING FACTOR FROM CALF THYMUS

IndoNLI: A Natural Language Inference Dataset for Indonesian

IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation

A Study of the State of the Art Approaches and Datasets for Multilingual Natural Language Inference

OCNLI: Original Chinese Natural Language Inference

IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages

NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages

A Survey on Recognizing Textual Entailment as an NLP Evaluation

IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languages

IndicDialogue: A dataset of subtitles in 10 Indic languages for Indic language modeling

A large annotated corpus for learning natural language inference

Natural Language Inference Using Lstm Model With Sentence Fusion

Identification of Entailment and Contradiction Relations between Natural Language Sentences: A Neurosymbolic Approach

IndicSTR12: A Dataset for Indic Scene Text Recognition

ArEntail: manually-curated Arabic natural language inference dataset from news headlines

IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages

IndoSum: A New Benchmark Dataset for Indonesian Text Summarization

Learning to Infer from Unlabeled Data: A Semi-supervised Learning Approach for Robust Natural Language Inference

Reading and Thinking: Re-read LSTM Unit for Textual Entailment Recognition.