Abstract:Annotation noise is widespread in datasets, but manually revising a flawed corpus is time-consuming and error-prone. Hence, given the prior knowledge in Pre-trained Language Models and the expected uniformity across all annotations, we attempt to reduce annotation noise in the corpus through two tasks automatically: (1) Annotation Inconsistency Detection that indicates the credibility of annotations, and (2) Annotation Error Correction that rectifies the abnormal annotations. We investigate how to acquire semantic sensitive annotation representations from Pre-trained Language Models, expecting to embed the examples with identical annotations to the mutually adjacent positions even without fine-tuning. We proposed a novel credibility score to reveal the likelihood of annotation inconsistencies based on the neighbouring consistency. Then, we fine-tune the Pre-trained Language Models based classifier with cross-validation for annotation correction. The annotation corrector is further elaborated with two approaches: (1) soft labelling by Kernel Density Estimation and (2) a novel distant-peer contrastive loss. We study the re-annotation in relation extraction and create a new manually revised dataset, Re-DocRED, for evaluating document-level re-annotation. The proposed credibility scores show promising agreement with human revisions, achieving a Binary F1 of 93.4 and 72.5 in detecting inconsistencies on TACRED and DocRED respectively. Moreover, the neighbour-aware classifiers based on distant-peer contrastive learning and uncertain labels achieve Macro F1 up to 66.2 and 57.8 in correcting annotations on TACRED and DocRED respectively. These improvements are not merely theoretical: Rather, automatically denoised training sets demonstrate up to 3.6% performance improvement for state-of-the-art relation extraction models.

Noise Correction on Subjective Datasets

Rethinking Noisy Label Learning in Real-world Annotation Scenarios from the Noise-type Perspective

OT Cleaner: Label Correction As Optimal Transport

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations

Leveraging Annotator Disagreement for Text Classification

Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks

Pre-trained Language Models as Re-Annotators

Three-way Decision-Based Noise Correction for Crowdsourcing

Learning to Segment from Noisy Annotations: A Spatial Correction Approach

Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning

Disjoint Contrastive Regression Learning for Multi-Sourced Annotations

Towards Noise-resistant Object Detection with Noisy Annotations

An joint end-to-end framework for learning with noisy labels

Meta Label Correction for Noisy Label Learning

Modeling Noisy Annotations for Point-wise Supervision

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?

Correct Twice at Once: Learning to Correct Noisy Labels for Robust Deep Learning

Learning to Purify Noisy Labels Via Meta Soft Label Corrector.

Learn2Agree: Fitting with Multiple Annotators without Objective Ground Truth