Abstract:Some complex problems, such as image tagging and natural language processing, are very challenging for computers, where even state-of-the-art technology is yet able to provide satisfactory accuracy. Therefore, rather than relying solely on developing new and better algorithms to handle such tasks, we look to the crowdsourcing solution -- employing human participation -- to make good the shortfall in current technology. Crowdsourcing is a good supplement to many computer tasks. A complex job may be divided into computer-oriented tasks and human-oriented tasks, which are then assigned to machines and humans respectively. To leverage the power of crowdsourcing, we design and implement a Crowdsourcing Data Analytics System, CDAS. CDAS is a framework designed to support the deployment of various crowdsourcing applications. The core part of CDAS is a quality-sensitive answering model, which guides the crowdsourcing engine to process and monitor the human tasks. In this paper, we introduce the principles of our quality-sensitive model. To satisfy user required accuracy, the model guides the crowdsourcing query engine for the design and processing of the corresponding crowdsourcing jobs. It provides an estimated accuracy for each generated result based on the human workers' historical performances. When verifying the quality of the result, the model employs an online strategy to reduce waiting time. To show the effectiveness of the model, we implement and deploy two analytics jobs on CDAS, a twitter sentiment analytics job and an image tagging job. We use real Twitter and Flickr data as our queries respectively. We compare our approaches with state-of-the-art classification and image annotation techniques. The results show that the human-assisted methods can indeed achieve a much higher accuracy. By embedding the quality-sensitive model into crowdsourcing query engine, we effectiv...[truncated].

Recovering Missing Labels of Crowdsourcing Workers.

Learning from Crowds under Experts' Supervision

A Formalized Framework for Incorporating Expert Labels in Crowdsourcing Environment

Towards Open-Set Semantic Labeling in 3D Point Clouds : Analysis on the Unknown Class.

Hierarchical Crowdsourcing for Data Labeling with Heterogeneous Crowd.

Distribution-Aware Crowdsourced Entity Collection

Crowdsourcing Label Quality: A Theoretical Analysis

Collusion Detection and Ground Truth Inference in Crowdsourcing for Labeling Tasks.

Unsupervised Crowdsourcing with Accuracy and Cost Guarantees

Uncovering the Latent Structures of Crowd Labeling.

Cost-efficient Crowdsourcing for Span-based Sequence Labeling: Worker Selection and Data Augmentation

Obtaining High-Quality Label by Distinguishing Between Easy and Hard Items in Crowdsourcing

Attention-Aware Answers of the Crowd

CDAS: A Crowdsourcing Data Analytics System

Labelling Training Samples Using Crowdsourcing Annotation for Recommendation

Truth Discovery in Sequence Labels from Crowds

Crowdsourced POI Labelling: Location-aware Result Inference and Task Assignment.

A Crowdsourcing Framework for Collecting Tabular Data

Selective Verification Strategy for Learning from Crowds

An Online Learning Approach to Improving the Quality of Crowd-Sourcing

Crowd-Selection Query Processing in Crowdsourcing Databases: A Task-Driven Approach.