Abstract:Crowdsourcing has recently become popular among machine learning researchers and social scientists as an effective way to collect large-scale experimental data from distributed workers. To extract useful information from the cheap but potentially unreliable answers to tasks, a key problem is to identify reliable workers as well as unambiguous tasks. Although for objective tasks that have one correct answer per task, previous works can estimate worker reliability and task clarity based on the single gold standard assumption, for tasks that are subjective and accept multiple reasonable answers that workers may be grouped into, a phenomenon called schools of thought, existing models cannot be trivially applied. In this work, we present a statistical model to estimate worker reliability and task clarity without resorting to the single gold standard assumption. This is instantiated by explicitly characterizing the grouping behavior to form schools of thought with a rank-1 factorization of a worker-task groupsize matrix. Instead of performing an intermediate inference step, which can be expensive and unstable, we present an algorithm to analytically compute the sizes of different groups. We perform extensive empirical studies on real data collected from Amazon Mechanical Turk. Our method discovers the schools of thought, shows reasonable estimation of worker reliability and task clarity, and is robust to hyperparameter changes. Furthermore, our estimated worker reliability can be used to improve the gold standard prediction for objective tasks.

Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction

Adaptive Crowdsourcing Via Self-Supervised Learning

The Importance of Being Earnest in Crowdsourcing Systems

Learning from Crowds in the Presence of Schools of Thought.

Incentivizing Evaluation via Limited Access to Ground Truth: Peer-Prediction Makes Things Worse

Full Characterization of Adaptively Strong Majority Voting in Crowdsourcing

Crowdsourcing with Difficulty: A Bayesian Rating Model for Heterogeneous Items

Icrowd: An Adaptive Crowdsourcing Framework

Strategic Information Revelation in Crowdsourcing Systems Without Verification

Globally Optimal Crowdsourcing Quality Management

Unsupervised Crowdsourcing with Accuracy and Cost Guarantees

A Collaborative Mechanism for Crowdsourcing Prediction Problems

Multicategory Crowdsourcing Accounting for Plurality in Worker Skill and Intention, Task Difficulty, and Task Heterogeneity

Learning to Predict the Wisdom of Crowds

Crowdsourcing with Unsure Option

Crowdsourcing in the Absence of Ground Truth -- A Case Study

Crowdsourced Outcome Determination in Prediction Markets

Peer Prediction with Heterogeneous Tasks

Mitigating Cognitive Biases in Multi-Criteria Crowd Assessment

CrowdGrader: Crowdsourcing the Evaluation of Homework Assignments

Dropout Prediction in Crowdsourcing Markets