What problem does this paper attempt to address?

This paper attempts to address the problem of how to quantify and evaluate the certainty and uncertainty of predictions in multi-class decision machine learning problems. Specifically, the authors propose two new metrics—certainty and doubt—that can be used within both Bayesian and frequentist frameworks to assess and compare the quality and uncertainty of predictions. ### Background and Motivation With the widespread application of artificial intelligence and machine learning technologies in critical safety applications, the demand for automating, accelerating, and enhancing decision-making processes is increasing. These technologies enable decision-makers to gain information and decision advantages at machine speed. However, in critical safety decisions, it is crucial to estimate and consider the levels of certainty and doubt for each classification decision. Even the least likely risk outcomes can significantly impact classification decisions, thus critical applications need to be sensitive to these tail probabilities. ### Main Contributions 1. **Definition of Certainty and Doubt**: - The authors define pairwise certainty and pairwise doubt and provide their mathematical expressions. - Certainty and doubt are defined as reciprocals of each other, with their geometric intuition explained in subsequent sections. - Raw certainty and raw doubt are also defined, which can be directly calculated from the probability vector output of the model. 2. **Geometric Interpretation**: - Certainty and doubt are naturally represented through projective geometry methods. - Using the concepts of projective space and projective geometry, certainty and doubt are mapped onto a projective line, thereby better understanding their properties. 3. **Cost Function**: - A cost function based on certainty and doubt is proposed to avoid situations where the model lacks confidence between different labels. - The design of the cost function considers the inherent smoothness of the model, ensuring reasonable penalization of doubt during the optimization process. ### Objective By introducing these new metrics, the authors aim to provide a more intuitive and effective evaluation method for multi-class decision problems, thereby improving the reliability and robustness of models in critical safety applications.

Measuring Classification Decision Certainty and Doubt

An Improved Belief Entropy and Its Application in Decision-Making

Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rules

A Meta-heuristic Approach to Estimate and Explain Classifier Uncertainty

Calibrating Expressions of Certainty

On Information-Theoretic Measures of Predictive Uncertainty

Legitimate ground-truth-free metrics for deep uncertainty classification scoring

Uncertainty quantification in graph-based classification of high dimensional data

Evaluating and Boosting Uncertainty Quantification in Classification

Towards More Accurate Uncertainty Estimation in Text Classification

The Certainty Ratio $C_ρ$: a novel metric for assessing the reliability of classifier predictions

Leveraging Uncertainty Estimates To Improve Classifier Performance

Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty

Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty?

Quantifying Uncertainty in Deep Learning Classification with Noise in Discrete Inputs for Risk-Based Decision Making

Theoretical characterization of uncertainty in high-dimensional linear classification

A Novel Measurement Of Uncertain Information Based On Interval Type-2 Cross-Entropy And Its Application

Scoring Rules and Calibration for Imprecise Probabilities

Identifying Incorrect Classifications with Balanced Uncertainty

Representing uncertainty and imprecision in machine learning: A survey on belief functions

Inconclusive quantum measurements and decisions under uncertainty