Abstract:In general, a large amount of labels are needed for supervised learning algorithms to achieve satisfactory performance. It's typically very time-consuming and money-consuming to get such kind of labeled data. Recently, crowdsourcing services provide an effective way to collect labeled data with much lower cost. Hence, crowdsourced learning (CL), which performs learning with labeled data collected from crowdsourcing services, has become a very hot and interesting research topic in recent years. Most existing CL methods exploit only the labels from different workers (annotators) for learning while ignoring the attributes of the instances. In many real applications, the attributes of the instances are actually the most discriminative information for learning. Hence, CL methods with attributes have attracted more and more attention from CL researchers. One representative model of such kind is the personal classifier (PC) model, which has achieved the state-of-the-art performance. However, the PC model makes an unreasonable assumption that all the workers contribute equally to the final classification. This contradicts the fact that different workers have different quality (ability) for data labeling. In this paper, we propose a novel model, called robust personal classifier (RPC), for robust crowdsourced learning. Our model can automatically learn an expertise score for each worker. This expertise score reflects the inherent quality of each worker. The final classifier of our RPC model gives high weights for good workers and low weights for poor workers or spammers, which is more reasonable than PC model with equal weights for all workers. Furthermore, the learned expertise score can be used to eliminate spammers or low-quality workers. Experiments on simulated datasets and UCI datasets show that the proposed model can dramatically outperform the baseline models such as PC model in terms of classification accuracy and ability to detect spammers.

SPGLAD: A Self-paced Learning-Based Crowdsourcing Classification Model.

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Learning from Crowds under Experts' Supervision

A Formalized Framework for Incorporating Expert Labels in Crowdsourcing Environment

Human-centred Design on Crowdsourcing Annotation Towards Improving Active Learning Model Performance

Quality Control for Crowdsourcing with Spatial and Temporal Distribution.

An Online Learning Approach to Improving the Quality of Crowd-Sourcing

Crowdsourcing with Self-paced Workers

Quality Control of Massive Data for Crowdsourcing in Location-Based Services.

Robust Crowdsourced Learning

Self-paced annotations of crowd workers

CDAS: A Crowdsourcing Data Analytics System

Deep Active Learning Framework for Crowdsourcing-Enhanced Image Classification and Segmentation

Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions

Integrating Crowdsourcing and Active Learning for Classification of Work-Life Events from Tweets

A Dynamic Difficulty-Sensitive Worker Distribution Model for Crowdsourcing Quality Management.

Crowdsourcing with Enhanced Data Quality Assurance: An Efficient Approach to Mitigate Resource Scarcity Challenges in Training Large Language Models for Healthcare

Obtaining High-Quality Label by Distinguishing Between Easy and Hard Items in Crowdsourcing

Application Research of Iterative Detection Strategy in the Crowdsourcing Quality Evaluation

Active Visual Recognition with Expertise Estimation in Crowdsourcing

A Joint Gaussian Process Model for Active Visual Recognition with Expertise Estimation in Crowdsourcing.