Abstract:Representation learning has been proven to play an important role in the unprecedented success of machine learning models in numerous tasks, such as machine translation, face recognition and recommendation. The majority of existing representation learning approaches often require a large number of consistent and noise-free labels. However, due to various reasons such as budget constraints and privacy concerns, labels are very limited in many real-world scenarios. Directly applying standard representation learning approaches on small labeled data sets will easily run into over-fitting problems and lead to sub-optimal solutions. Even worse, in some domains such as education, the limited labels are usually annotated by multiple workers with diverse expertise, which yields noises and inconsistency in such crowdsourcing settings. In this paper, we propose a novel framework which aims to learn effective representations from limited data with crowdsourced labels. Specifically, we design a grouping based deep neural network to learn embeddings from a limited number of training samples and present a Bayesian confidence estimator to capture the inconsistency among crowdsourced labels. Furthermore, to expedite the training process, we develop a hard example selection procedure to adaptively pick up training examples that are misclassified by the model. Extensive experiments conducted on three real-world data sets demonstrate the superiority of our framework on learning representations from limited data with crowdsourced labels, comparing with various state-of-the-art baselines. In addition, we provide a comprehensive analysis on each of the main components of our proposed framework and also introduce the promising results it achieved in our real production to fully understand the proposed framework. To encourage reproducible results, we make our code available online at https://github.com/tal-ai/RECLE .

Learning from biased crowdsourced labeling with deep clustering

Learning from Crowds under Experts' Supervision

A Formalized Framework for Incorporating Expert Labels in Crowdsourcing Environment

Recovering Missing Labels of Crowdsourcing Workers.

Consensus Algorithms for Biased Labeling in Crowdsourcing.

Using Cluster Information to Improve Label Propagation

Hierarchical Crowdsourcing for Data Labeling with Heterogeneous Crowd.

Unbiased Multi-Label Learning from Crowdsourced Annotations

Deep Fair Discriminative Clustering

SemCluster: a Semi-Supervised Clustering Tool for Crowdsourced Test Reports with Deep Image Understanding.

Optimizing the Wisdom of the Crowd: Inference, Learning, and Teaching

Semi-crowdsourced Clustering with Deep Generative Models

Modeling for Noisy Labels of Crowd Workers.

Crowdsourcing Label Quality: A Theoretical Analysis

Uncovering the Latent Structures of Crowd Labeling.

Reducing Spatial Labeling Redundancy for Semi-supervised Crowd Counting

Self-Taught Active Learning from Crowds

Representation Learning from Limited Educational Data with Crowdsourced Labels

Labeling Clusters from Both Linguistic and Statistical Perspectives: A Hybrid Approach

End-to-End Annotator Bias Approximation on Crowdsourced Single-Label Sentiment Analysis

Labelling Training Samples Using Crowdsourcing Annotation for Recommendation