Abstract:Crowdsourcing systems commonly face the problem of aggregating multiple judgments provided by potentially unreliable workers. In addition, several aspects of the design of efficient crowdsourcing processes, such as defining worker's bonuses, fair prices and time limits of the tasks, involve knowledge of the likely duration of the task at hand. Bringing this together, in this work we introduce a new time--sensitive Bayesian aggregation method that simultaneously estimates a task's duration and obtains reliable aggregations of crowdsourced judgments. Our method, called BCCTime, builds on the key insight that the time taken by a worker to perform a task is an important indicator of the likely quality of the produced judgment. To capture this, BCCTime uses latent variables to represent the uncertainty about the workers' completion time, the tasks' duration and the workers' accuracy. To relate the quality of a judgment to the time a worker spends on a task, our model assumes that each task is completed within a latent time window within which all workers with a propensity to genuinely attempt the labelling task (i.e., no spammers) are expected to submit their judgments. In contrast, workers with a lower propensity to valid labeling, such as spammers, bots or lazy labelers, are assumed to perform tasks considerably faster or slower than the time required by normal workers. Specifically, we use efficient message-passing Bayesian inference to learn approximate posterior probabilities of (i) the confusion matrix of each worker, (ii) the propensity to valid labeling of each worker, (iii) the unbiased duration of each task and (iv) the true label of each task. Using two real-world public datasets for entity linking tasks, we show that BCCTime produces up to 11% more accurate classifications and up to 100% more informative estimates of a task's duration compared to state-of-the-art methods.

Transitivity, Time Consumption, and Quality of Preference Judgments in Crowdsourcing

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably

Crowdsourced Top- K Queries by Pairwise Preference Judgments with Confidence and Budget Control

Context-based Collective Preference Aggregation for Prioritizing Crowd Opinions in Social Decision-making

Crowd-Selection Query Processing in Crowdsourcing Databases: A Task-Driven Approach.

Preference-aware Task Assignment in Spatial Crowdsourcing: from Individuals to Groups

Preference-aware Group Task Assignment in Spatial Crowdsourcing: Effectiveness and Efficiency

Aggregating Quantitative Relative Judgments: From Social Choice to Ranking Prediction

Crowdsourcing and Evaluating Text-Based Audio Retrieval Relevances

Select-the-Best-Ones: A new way to judge relative relevance

Listwise Approach For Rank Aggregation In Crowdsourcing

Impact of the Number of Votes on the Reliability and Validity of Subjective Speech Quality Assessment in the Crowdsourcing Approach

Bilateral Preference-aware Task Assignment in Spatial Crowdsourcing

Mitigating Cognitive Biases in Multi-Criteria Crowd Assessment

Task Assignment with Efficient Federated Preference Learning in Spatial Crowdsourcing

Time-Sensitive Bayesian Information Aggregation for Crowdsourcing Systems

Crowdsourcing in the Absence of Ground Truth -- A Case Study

Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing

Automated Quality Assessment for Crowdsourced Test Reports Based on Dependency Parsing

Hear Me Out: A Study on the Use of the Voice Modality for Crowdsourced Relevance Assessments