Abstract:Given a learning problem with real-world tradeoffs, which cost function should the model be trained to optimize? This is the metric selection problem in machine learning. Despite its practical interest, there is limited formal guidance on how to select metrics for machine learning applications. This thesis outlines metric elicitation as a principled framework for selecting the performance metric that best reflects implicit user preferences. Once specified, the evaluation metric can be used to compare and train models. In this manuscript, we formalize the problem of Metric Elicitation and devise novel strategies for eliciting classification performance metrics using pairwise preference feedback over classifiers. Specifically, we provide novel strategies for eliciting linear and linear-fractional metrics for binary and multiclass classification problems, which are then extended to a framework that elicits group-fair performance metrics in the presence of multiple sensitive groups. All the elicitation strategies that we discuss are robust to both finite sample and feedback noise, thus are useful in practice for real-world applications. Using the tools and the geometric characterizations of the feasible confusion statistics sets from the binary, multiclass, and multiclass-multigroup classification setups, we further provide strategies to elicit from a wider range of complex, modern multiclass metrics defined by quadratic functions of confusion statistics by exploiting their local linear structure. From application perspective, we also propose to use the metric elicitation framework in optimizing complex black box metrics that is amenable to deep network training. Lastly, to bring theory closer to practice, we conduct a preliminary real-user study that shows the efficacy of the metric elicitation framework in recovering the users' preferred performance metric in a binary classification setup.

CORRELATION ANALYSIS OF PERFORMANCE METRICS FOR CLASSIFIER

A Strategy on Selecting Performance Metrics for Classifier Evaluation.

Analysis and Comparison of Classification Metrics

Empirical analysis of performance assessment for imbalanced classification

Selecting a classification performance measure: matching the measure to the problem

A unifying view for performance measures in multi-class prediction

Classification Performance Metric Elicitation and its Applications

Cost-sensitive performance metric for comparing multiple ordinal classifiers

Metrics for Multi-Class Classification: an Overview

Is Correlation Ranking Really Reliable for the Performance Counter Selection Conducted for Power Estimation?

Why Cohen’s Kappa should be avoided as performance measure in classification

Multi-Level Comparison of Machine Learning Classifiers and Their Performance Metrics

Measuring Class-Imbalance Sensitivity of Deterministic Performance Evaluation Metrics

Performance Metrics for Probabilistic Ordinal Classifiers

Performance Analysis and Comparison of Neural Networks and Support Vector Machines Classifier

Does the evaluation stand up to evaluation? A first-principle approach to the evaluation of classifiers

Comparing discriminating abilities of evaluation metrics in link prediction

Exploring Symmetry of Binary Classification Performance Metrics

BenchMetrics: a systematic benchmarking method for binary classification performance metrics