Abstract:Abstract Against the backdrop of the ever-improving Natural Language Inference (NLI) models, recent efforts have focused on the suitability of the current NLI datasets and on the feasibility of the NLI task as it is currently approached. Many of the recent studies have exposed the inherent human disagreements of the inference task and have proposed a shift from categorical labels to human subjective probability assessments, capturing human uncertainty. In this work, we show how neither the current task formulation nor the proposed uncertainty gradient are entirely suitable for solving the NLI challenges. Instead, we propose an ordered sense space annotation, which distinguishes between logical and common-sense inference. One end of the space captures non-sensical inferences, while the other end represents strictly logical scenarios. In the middle of the space, we find a continuum of common-sense, namely, the subjective and graded opinion of a “person on the street.” To arrive at the proposed annotation scheme, we perform a careful investigation of the SICK corpus and we create a taxonomy of annotation issues and guidelines. We re-annotate the corpus with the proposed annotation scheme, utilizing four symbolic inference systems, and then perform a thorough evaluation of the scheme by fine-tuning and testing commonly used pre-trained language models on the re-annotated SICK within various settings. We also pioneer a crowd annotation of a small portion of the MultiNLI corpus, showcasing that it is possible to adapt our scheme for annotation by non-experts on another NLI corpus. Our work shows the efficiency and benefits of the proposed mechanism and opens the way for a careful NLI task refinement.

A Two-Stage Incremental Annotation Approach to Constructing A Network Informal Language Corpus

Normalization of Chinese Chat Language

Research on Corpus Annotation Method Based on Collective Intelligence

Linguistic and Behavioural Studies of Chinese Chat Language

Universal Dependencies Parsing For Colloquial Singaporean English

The Construction of Chinese Multi-dimensional Learner Corpus:YACLC

Curing the SICK and Other NLI Maladies

Building Low-Resource NER Models Using Non-Speaker Annotation

AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

OCNLI: Original Chinese Natural Language Inference

Limpeh ga li gong: Challenges in Singlish Annotations

Asynchronous Deep Interaction Network for Natural Language Inference.

Applying Cooperative Machine Learning to Speed Up the Annotation of Social Signals in Large Multi-modal Corpora

YACLC: A Chinese Learner Corpus with Multidimensional Annotation

CollabKG: A Learnable Human-Machine-Cooperative Information Extraction Toolkit for (Event) Knowledge Graph Construction

Toward Anomalous and Dynamic Nature of the Chinese Network Chat Language

Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis

Implementing a Portable Clinical NLP System with a Common Data Model - a Lisp Perspective

A Multi-level and Multi-label Annotation Strategy for User Questions in ICT Customer Service

A Neural Span-Based Continual Named Entity Recognition Model

Networked Artificial Intelligence English Translation System Based on an Intelligent Knowledge Base and Translation Method Thereof