Abstract:Online crowdsourcing provides a scalable and inexpensive means to collect knowledge (e.g. labels) about various types of data items (e.g. text, audio, video). However, it is also known to result in large variance in the quality of recorded responses which often cannot be directly used for training machine learning systems. To resolve this issue, a lot of work has been conducted to control the response quality such that low-quality responses cannot adversely affect the performance of the machine learning systems. Such work is referred to as the quality control for crowdsourcing. Past quality control research can be divided into two major branches: quality control mechanism design and statistical models. The first branch focuses on designing measures, thresholds, interfaces and workflows for payment, gamification, question assignment and other mechanisms that influence workers' behaviour. The second branch focuses on developing statistical models to perform effective aggregation of responses to infer correct responses. The two branches are connected as statistical models (i) provide parameter estimates to support the measure and threshold calculation, and (ii) encode modelling assumptions used to derive (theoretical) performance guarantees for the mechanisms. There are surveys regarding each branch but they lack technical details about the other branch. Our survey is the first to bridge the two branches by providing technical details on how they work together under frameworks that systematically unify crowdsourcing aspects modelled by both of them to determine the response quality. We are also the first to provide taxonomies of quality control papers based on the proposed frameworks. Finally, we specify the current limitations and the corresponding future directions for the quality control research.

Exploring the trends, characteristic antecedents, and performance consequences of crowdsourcing project risks

Hybrid Influences of Social Subsystem and Technical Subsystem Risks in the Crowdsourcing Marketplace

An Incentive Mechanism for Crowdsourcing Systems with Network Effects

Imagery Based Risk Assessment Using Crowdsourcing Technology in Complex Workspaces

Study on the Influencing Factors of Task Performance of Crowdsourcing Platform Based on Contracting Method

Quality-Aware Incentive Mechanisms Under Social Influences in Data Crowdsourcing

Understanding consumers' continuous-use intention of crowdsourcing logistics services: Empirical evidence from China

Towards a characterization of crowdsourcing practices

Research on the Optimal Combination and Scheduling Method of Crowdsourcing Members in a Cloud Design Platform

Managing Risk in Business Centric Crowdfunding Platforms.

Factors Influencing the Bid‐Winning of Respondents in the Context of Crowdsourcing

Effects of Intrinsic and Extrinsic Motivational Factors on Employee Participation in Internal Crowdsourcing Initiatives in China

Understanding Fatigue and its Impact in Crowdsourcing

Crowdsourcing as a strategic IS sourcing phenomenon: Critical review and insights for future research

Risk Disclosure in Crowdfunding

Antecedents of Solution Quality in Crowdsourcing: the Sponsor'S Perspective

Foreign Research Status of Crowdsourcing

Information Disclosure and Crowdfunding: an Empirical Analysis of the Disclosure of Project Risk

A Bibliometric Analysis of Crowdsourcing in the Field of Public Health

A technical survey on statistical modelling and design methods for crowdsourcing quality control