Abstract:Microtask crowdsourcing is a form of crowdsourcing in which work is decomposed into a set of small, self-contained tasks, which each can typically be completed in a matter of minutes. Due to the various capabilities and knowledge background of the voluntary participants on the Internet, the answers collected from the crowd are ambiguous and the final answer aggregation is challenging. In this process, the choice of quality control strategies is important for ensuring the quality of the crowdsourcing results. Previous work on answer estimation mainly used expectation–maximization (EM) approach. Unfortunately, EM provides local optimal solutions and the estimated results will be affected by the initial value. In this paper, we extend the local optimal result of EM and propose an approximate global optimal algorithm for answer aggregation of crowdsourcing microtasks with binary answers. Our algorithm is expected to improve the accuracy of real answer estimation through further likelihood maximization. First, three worker quality evaluation models are presented based on static and dynamic methods, respectively, and the local optimal results are obtained based on the maximum likelihood estimation method. Then, a dominance ordering model (DOM) is proposed according to the known worker responses and worker categories for the specified crowdsourcing task to reduce the space of potential task-response sequence while retaining the dominant sequence. Subsequently, a Cut-point neighbor detection algorithm is designed to iteratively search for the approximate global optimal estimation in a reduced space, which works on the proposed dominance ordering model (DOM). We conduct extensive experiments on both simulated and real-world datasets, and the experimental results illustrate that the proposed approach can obtain better estimation results and has higher performance than regular EM-based algorithms.

Answer Aggregation of Crowdsourcing Employing an Improved EM-Based Approach.

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Answer Aggregation for Crowdsourcing Microtasks Using Approximate Global Optimal Searching.

Achieving Approximate Global Optimization of Truth Inference for Crowdsourcing Microtasks

A Dynamic Difficulty-Sensitive Worker Distribution Model for Crowdsourcing Quality Management.

Crowdmerge: Achieving Optimal Crowdsourcing Quality Management By Sequent Merger

An Error Consistency Based Approach to Answer Aggregation in Open-Ended Crowdsourcing

Multicategory Crowdsourcing Accounting for Plurality in Worker Skill and Intention, Task Difficulty, and Task Heterogeneity

Optimal Crowdsourcing Scheme for Open Questions on Web

Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations

Crowd-Answering System Via Microblogging

Globally Optimal Crowdsourcing Quality Management

Attention-Aware Answers of the Crowd

A Weighted Aggregation Rule in Crowdsourcing Systems for High Result Accuracy

Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?

A Pruned DOM-Based Iterative Strategy for Approximate Global Optimization in Crowdsourcing Microtasks.

Crowdsourcing in the Absence of Ground Truth -- A Case Study

A Subjectivity-Aware Algorithm for Label Aggregation in Crowdsourcing

Empirical Study On Assessment Algorithms With Confidence In Crowdsourcing

Learning from Crowds in the Presence of Schools of Thought.

Adaptive Result Inference for Collecting Quantitative Data with Crowdsourcing.