Abstract:Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without arguments, and blurry terminology invites misconceptions. For instance, many works use so-called 'macro' metrics to rank systems (e.g., 'macro F1') but do not clearly specify what they would expect from such a `macro' metric. This is problematic, since picking a metric can affect research findings, and thus any clarity in the process should be maximized. Starting from the intuitive concepts of bias and prevalence, we perform an analysis of common evaluation metrics. The analysis helps us understand the metrics' underlying properties, and how they align with expectations as found expressed in papers. Then we reflect on the practical situation in the field, and survey evaluation practice in recent shared tasks. We find that metric selection is often not supported with convincing arguments, an issue that can make a system ranking seem arbitrary. Our work aims at providing overview and guidance for more informed and transparent metric selection, fostering meaningful evaluation.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problems of unclear and lack of transparency in common evaluation practices in classification assessment. Specifically, the author points out the following key issues: 1. **Arbitrariness in the selection of evaluation metrics**: - Many studies lack sufficient reasons or arguments when choosing evaluation metrics, resulting in the selection of metrics appearing arbitrary. - Ambiguous terms (such as "macro - average") are prone to cause misunderstandings, making readers and authors misjudge the evaluation results. 2. **Inconsistency between metrics and expectations**: - Researchers often hope that evaluation metrics can reflect specific performance characteristics (such as balanced treatment of different classes), but it is not clear whether the actually used metrics really meet these expectations. - For example, many works use "macro - average F1" to rank systems, but do not clearly state what their specific expectations for this "macro - average" metric are. 3. **The impact of metric selection on research results**: - Selecting different evaluation metrics may significantly affect research conclusions, so it is necessary to maximize the clarity and transparency in the evaluation process. 4. **Providing more reasonable guidance for metric selection**: - The author hopes to help researchers better understand these metrics by analyzing the characteristics of common evaluation metrics and make more informed and transparent decisions when choosing evaluation metrics. ### Main objectives - **Clarify and define the characteristics of evaluation metrics**: By introducing five basic properties (monotonicity, class - sensitivity, class - decomposability, prior - invariance, and chance - correction), the author conducts a detailed analysis of common classification evaluation metrics. - **Reflect on current evaluation practices**: By investigating the evaluation practices in shared tasks in recent years, the problems existing in metric selection are revealed. - **Provide guidance and suggestions**: Provide practical guidelines for researchers on how to select and interpret evaluation metrics to promote more meaningful evaluation. ### Conclusion The paper emphasizes the factors that should be considered when selecting evaluation metrics and puts forward specific suggestions to ensure the accuracy and interpretability of evaluation results. In this way, the author hopes to improve the transparency and scientific nature of classification evaluation, thereby promoting the research progress in related fields.

A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice

We Need to Talk About Classification Evaluation Metrics in NLP

Ranking evaluation metrics from a group-theoretic perspective

Analysis and Comparison of Classification Metrics

Does the evaluation stand up to evaluation? A first-principle approach to the evaluation of classifiers

Classification Performance Metric Elicitation and its Applications

Metrics for Multi-Class Classification: an Overview

On Search Engine Evaluation Metrics

A Strategy on Selecting Performance Metrics for Classifier Evaluation.

A Unified Study of Machine Learning Explanation Evaluation Metrics

Overcoming Common Flaws in the Evaluation of Selective Classification Systems

Understanding metric-related pitfalls in image analysis validation

Performance Evaluation in Machine Learning: The Good, the Bad, the Ugly, and the Way Forward

Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory

Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics

A Unified View of Evaluation Metrics for Structured Prediction

Empirical analysis of performance assessment for imbalanced classification

A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification

Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations

Towards a guideline for evaluation metrics in medical image segmentation