Abstract:This paper considers the challenge of evaluating a set of classifiers, as done in shared task evaluations like the KDD Cup or NIST TREC, without expert labels. While expert labels provide the traditional cornerstone for evaluating statistical learners, limited or expensive access to experts represents a practical bottleneck. Instead, we seek methodology for estimating performance of the classifiers which is more scalable than expert labeling yet preserves high correlation with evaluation based on expert labels. We consider both: 1) using only labels automatically generated by the classifiers (blind evaluation); and 2) using labels obtained via crowdsourcing. While crowdsourcing methods are lauded for scalability, using such data for evaluation raises serious concerns given the prevalence of label noise. In regard to blind evaluation, two broad strategies are investigated: combine & score and score & combine methods infer a single pseudo-gold label set by aggregating classifier labels; classifiers are then evaluated based on this single pseudo-gold label set. On the other hand, score & combine methods: 1) sample multiple label sets from classifier outputs, 2) evaluate classifiers on each label set, and 3) average classifier performance across label sets. When additional crowd labels are also collected, we investigate two alternative avenues for exploiting them: 1) direct evaluation of classifiers; or 2) supervision of combine & score methods. To assess generality of our techniques, classifier performance is measured using four common classification metrics, with statistical significance tests. Finally, we measure both score and rank correlations between estimated classifier performance vs. actual performance according to expert judgments. Rigorous evaluation of classifiers from the TREC 2011 Crowdsourcing Track shows reliable evaluation can be achieved without reliance on expert labels.

Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts

Crowdsourcing in the Absence of Ground Truth -- A Case Study

Measuring the Expertise of Workers for Crowdsourcing Applications

Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?

Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions

Crowdsourcing and Evaluating Text-Based Audio Retrieval Relevances

Adaptive Crowdsourcing Via Self-Supervised Learning

The Importance of Being Earnest in Crowdsourcing Systems

Learning from Crowds in the Presence of Schools of Thought.

An exploratory study of L1-specific non-words

Listwise Approach For Rank Aggregation In Crowdsourcing

Competence Development in Crowdsourcing: Examining the Impact of Different Evaluation Standards.

Multicategory Crowdsourcing Accounting for Plurality in Worker Skill and Intention, Task Difficulty, and Task Heterogeneity

Evaluating Search System Explainability with Psychometrics and Crowdsourcing

Evaluating Classifiers Without Expert Labels

Answer Inference for Crowdsourcing Based Scoring

BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing

Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing

Optimizing positional scoring rules for rank aggregation

Style Over Substance: Evaluation Biases for Large Language Models

Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation