Abstract:This paper considers the challenge of evaluating a set of classifiers, as done in shared task evaluations like the KDD Cup or NIST TREC, without expert labels. While expert labels provide the traditional cornerstone for evaluating statistical learners, limited or expensive access to experts represents a practical bottleneck. Instead, we seek methodology for estimating performance of the classifiers which is more scalable than expert labeling yet preserves high correlation with evaluation based on expert labels. We consider both: 1) using only labels automatically generated by the classifiers (blind evaluation); and 2) using labels obtained via crowdsourcing. While crowdsourcing methods are lauded for scalability, using such data for evaluation raises serious concerns given the prevalence of label noise. In regard to blind evaluation, two broad strategies are investigated: combine & score and score & combine methods infer a single pseudo-gold label set by aggregating classifier labels; classifiers are then evaluated based on this single pseudo-gold label set. On the other hand, score & combine methods: 1) sample multiple label sets from classifier outputs, 2) evaluate classifiers on each label set, and 3) average classifier performance across label sets. When additional crowd labels are also collected, we investigate two alternative avenues for exploiting them: 1) direct evaluation of classifiers; or 2) supervision of combine & score methods. To assess generality of our techniques, classifier performance is measured using four common classification metrics, with statistical significance tests. Finally, we measure both score and rank correlations between estimated classifier performance vs. actual performance according to expert judgments. Rigorous evaluation of classifiers from the TREC 2011 Crowdsourcing Track shows reliable evaluation can be achieved without reliance on expert labels.

Ground-truth or DAER: Selective Re-query of Secondary Information

Objective Performance Evaluation of Video Segmentation Algorithms with Ground-Truth

Virtual Ground Truth, and Pre-selection of 3D Interest Points for Improved Repeatability Evaluation of 2D Detectors

You Never Get a Second Chance To Make a Good First Impression: Seeding Active Learning for 3D Semantic Segmentation

Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding

PARSEG: a computationally efficient approach for statistical validation of botanical seeds' images

Rethinking Training Objective for Self-Supervised Monocular Depth Estimation - Semantic Cues to Rescue.

Uncovering the Full Potential of Visual Grounding Methods in VQA

Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales

Re-thinking Knowledge Graph Completion Evaluation from an Information Retrieval Perspective

SEED: A Simple and Effective 3D DETR in Point Clouds

Tracking Object Positions in Reinforcement Learning: A Metric for Keypoint Detection (extended version)

Grounding Consistency: Distilling Spatial Common Sense for Precise Visual Relationship Detection

Selective Verification Strategy for Learning from Crowds

Identification of Fine-grained Systematic Errors via Controlled Scene Generation

Visual Ground Truth Construction as Faceted Classification

An Empirical Study of Automated Mislabel Detection in Real World Vision Datasets

Perception Quality Evaluation with Visual and Infrared Cameras in Challenging Environmental Conditions

Evaluating Classifiers Without Expert Labels

Seed Classification using Synthetic Image Datasets Generated from Low-Altitude UAV Imagery

Seeded region growing: an extensive and comparative study