Abstract:The problem of crowdsourced entity collection solicits people (a.k.a. workers) to complete missing data in a database and has witnessed many applications in knowledge base completion and enterprise data collection. Although previous studies have attempted to address the "open world" challenge of crowdsourced entity collection, they do not pay much attention to the "distribution" of the collected entities. Evidently, in many real applications, users may have distribution requirements on the collected entities, e.g., even spatial distribution when collecting points-of-interest. In this paper, we study a new research problem, distribution-aware crowdsourced entity collection (CROWDDEC): Given an expected distribution w.r.t. an attribute (e.g., region or year), it aims to collect a set of entities via crowdsourcing and minimize the difference of the entity distribution from the expected distribution. Due to the openness of crowdsourcing, the CROWDDEC problem calls for effective crowdsourcing quality control. We propose an adaptive worker selection approach to address this problem. The approach estimates underlying entity distribution of workers on-the-fly based on the collected entities. Then, it adaptively selects the best set of workers that minimizes the difference from the expected distribution. Once workers submit their answers, it adjusts the estimation of workers' underlying distributions for subsequent adaptive worker selections. We prove the hardness of the problem, and develop effective estimation techniques as well as efficient worker selection algorithms to support this approach. We deployed the proposed approach on Amazon Mechanical Turk and the experimental results on two real datasets show that the approach achieves superiority on both effectiveness and efficiency.

Dropout Prediction in Crowdsourcing Markets

Stable Job Assignment for Crowdsourcing.

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Online Trichromatic Pickup and Delivery Scheduling in Spatial Crowdsourcing

Adaptive Crowdsourcing Via Self-Supervised Learning

Distribution-Aware Crowdsourced Entity Collection

A Collaborative Mechanism for Crowdsourcing Prediction Problems

Decision Support for Increasing the Efficiency of Crowdsourced Software Development

Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction

Dropout Model Evaluation in MOOCs

Crowdsourced Outcome Determination in Prediction Markets

Greedy Scheduling: A Neural Network Method to Reduce Task Failure in Software Crowdsourcing

Ensemble Deep Learning Network Model for Dropout Prediction in MOOCs

Evaluating Collaborative and Autonomous Agents in Data-Stream-Supported Coordination of Mobile Crowdsourcing

An online cost sensitive decision-making method in crowdsourcing systems.

Unsupervised Crowdsourcing with Accuracy and Cost Guarantees

Behavior Dynamics of Multiple Crowdsourcers in Mobile Crowdsourcing Markets

Machine Learning Application in Moocs: Dropout Prediction

Minimizing Entropy for Crowdsourcing with Combinatorial Multi-Armed Bandit

Icrowd: An Adaptive Crowdsourcing Framework

Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?