Abstract:Increased access to high-resolution topography has revolutionized our ability to map out fine-scale topographic features at watershed to landscape scales. As our "vision" of the land surface has improved, so has the need for more robust quantification of the accuracy of the geomorphic maps we derive from these data. One broad class of mapping challenges is that of binary classification whereby remote sensing data are used to identify the presence or absence of a given feature. Fortunately, there is a large suite of metrics developed in the data sciences well suited to quantifying the pixel-level accuracy of binary classifiers. This analysis focuses on how these metrics perform when there is a need to quantify how the number and extent of landforms are expected to vary as a function of the environmental forcing (e.g., due to climate, ecology, material property, erosion rate). Results from a suite of synthetic surfaces show how the most widely used pixel-level accuracy metric, the F 1 score, is particularly poorly suited to quantifying accuracy for this kind of application. Well-known biases to imbalanced data are exacerbated by methodological strategies that calibrate and validate classifiers across settings where feature abundances vary. The Matthews correlation coefficient largely removes this bias over a wide range of feature abundances such that the sensitivity of accuracy scores to geomorphic setting instead embeds information about the size and shape of features and the type of error. If error is random, the Matthews correlation coefficient is insensitive to feature size and shape, though preferential modification of the dominant class can limit the domain over which scores can be compared. If the error is systematic (e.g., due to co-registration error between remote sensing datasets), this metric shows strong sensitivity to feature size and shape such that smaller features with more complex boundaries induce more classification error. Future studies should build on this analysis by interrogating how pixel-level accuracy metrics respond to different kinds of feature distributions indicative of different types of surface processes.

Generalized Approach for Using Unbiased Symmetric Metrics with Negative Values: Normalized Mean Bias Factor and Normalized Mean Absolute Error Factor

New unbiased symmetric metrics for evaluation of air quality models

De-biasing "bias" measurement

A Principled Approach for a New Bias Measure

Normalized mutual information is a biased measure for classification and community detection

Developing a Dataset-Adaptive, Normalized Metric for Machine Learning Model Assessment: Integrating Size, Complexity, and Class Imbalance

Measuring Class-Imbalance Sensitivity of Deterministic Performance Evaluation Metrics

Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition

New Robust Metrics of Central Tendency for Estimation Performance Evaluation

MAD (about median) vs. quantile-based alternatives for classical standard deviation, skewness, and kurtosis

A better measure of relative prediction accuracy for model selection and model estimation

New Concentration Metrics for Performance Evaluation of Estimation Algorithms.

Normalization of zero-inflated data: An empirical analysis of a new indicator family and its use with altmetrics data

Different Bias Under Different Criteria: Assessing Bias in LLMs with a Fact-Based Approach

Empirical analysis of performance assessment for imbalanced classification

Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies

Normalization of zero-inflated data: An empirical analysis of a new indicator family

Evaluating the accuracy of binary classifiers for geomorphic applications

Evaluating Metrics for Bias in Word Embeddings

Geometric mean method for judgement matrices: Formulas for errors

Generalizing the normality: a novel towards different estimation methods for skewed information