Abstract:As the size of the datasets getting larger, accurately annotating suchdatasets is becoming more impractical due to the expensiveness on both time andeconomy. Therefore, crowd-sourcing has been widely adopted to alleviate thecost of collecting labels, which also inevitably introduces label noise andeventually degrades the performance of the model. To learn from crowd-sourcingannotations, modeling the expertise of each annotator is a common butchallenging paradigm, because the annotations collected by crowd-sourcing areusually highly-sparse. To alleviate this problem, we propose Coupled ConfusionCorrection (CCC), where two models are simultaneously trained to correct theconfusion matrices learned by each other. Via bi-level optimization, theconfusion matrices learned by one model can be corrected by the distilled datafrom the other. Moreover, we cluster the “annotator groups” who share similarexpertise so that their confusion matrices could be corrected together. In thisway, the expertise of the annotators, especially of those who provide seldomlabels, could be better captured. Remarkably, we point out that the annotationsparsity not only means the average number of labels is low, but also there arealways some annotators who provide very few labels, which is neglected byprevious works when constructing synthetic crowd-sourcing annotations. Based onthat, we propose to use Beta distribution to control the generation of thecrowd-sourcing labels so that the synthetic annotations could be moreconsistent with the real-world ones. Extensive experiments are conducted on twotypes of synthetic datasets and three real-world datasets, the results of whichdemonstrate that CCC significantly outperforms state-of-the-art approaches.Source codes are available at: https://github.com/Hansong-Zhang/CCC.

Homophily-aware correction framework for crowdsourced labels using heterogeneous information network

A Formalized Framework for Incorporating Expert Labels in Crowdsourcing Environment

Learning from Crowds under Experts' Supervision

Recovering Missing Labels of Crowdsourcing Workers.

OT Cleaner: Label Correction As Optimal Transport

Hierarchical Crowdsourcing for Data Labeling with Heterogeneous Crowd.

Coupled Confusion Correction: Learning from Crowds with Sparse Annotations

Improving Crowdsourced Label Quality Using Noise Correction.

Collusion Detection and Ground Truth Inference in Crowdsourcing for Labeling Tasks.

Uncovering the Latent Structures of Crowd Labeling.

Certainty weighted voting-based noise correction for crowdsourcing

Optimizing the Wisdom of the Crowd: Inference, Learning, and Teaching

Learn from Heterophily: Heterophilous Information-enhanced Graph Neural Network

Three-way Decision-Based Noise Correction for Crowdsourcing

Crowdsourced POI Labelling: Location-aware Result Inference and Task Assignment.

SiMilarity-Enhanced Homophily for Multi-View Heterophilous Graph Clustering

Exploiting Heterogeneous Graph Neural Networks with Latent Worker/Task Correlation Information for Label Aggregation in Crowdsourcing

Heterophily-Aware Graph Attention Network

HOFA: Twitter Bot Detection with Homophily-Oriented Augmentation and Frequency Adaptive Attention

Hetero^2Net: Heterophily-aware Representation Learning on Heterogenerous Graphs