Abstract:Increased access to high-resolution topography has revolutionized our ability to map out fine-scale topographic features at watershed to landscape scales. As our "vision" of the land surface has improved, so has the need for more robust quantification of the accuracy of the geomorphic maps we derive from these data. One broad class of mapping challenges is that of binary classification whereby remote sensing data are used to identify the presence or absence of a given feature. Fortunately, there is a large suite of metrics developed in the data sciences well suited to quantifying the pixel-level accuracy of binary classifiers. This analysis focuses on how these metrics perform when there is a need to quantify how the number and extent of landforms are expected to vary as a function of the environmental forcing (e.g., due to climate, ecology, material property, erosion rate). Results from a suite of synthetic surfaces show how the most widely used pixel-level accuracy metric, the F 1 score, is particularly poorly suited to quantifying accuracy for this kind of application. Well-known biases to imbalanced data are exacerbated by methodological strategies that calibrate and validate classifiers across settings where feature abundances vary. The Matthews correlation coefficient largely removes this bias over a wide range of feature abundances such that the sensitivity of accuracy scores to geomorphic setting instead embeds information about the size and shape of features and the type of error. If error is random, the Matthews correlation coefficient is insensitive to feature size and shape, though preferential modification of the dominant class can limit the domain over which scores can be compared. If the error is systematic (e.g., due to co-registration error between remote sensing datasets), this metric shows strong sensitivity to feature size and shape such that smaller features with more complex boundaries induce more classification error. Future studies should build on this analysis by interrogating how pixel-level accuracy metrics respond to different kinds of feature distributions indicative of different types of surface processes.

Spatial classification in the presence of measurement error

Classification under Data Contamination with Application to Remote Sensing Image Mis-registration

On the Sampling Strategy for Evaluation of Spectral-Spatial Methods in Hyperspectral Image Classification

Spatial Measurement Errors in the Field of Spatial Epidemiology

Consistent Validation for Predictive Methods in Spatial Settings

Spatial analysis of remote sensing image classification accuracy

Estimating the prediction performance of spatial models via spatial k-fold cross validation

On the Sampling Strategies for Evaluation of Joint Spectral-Spatial Information Based Classifiers

Likelihood-based Estimation of Spatial Intensity and Variation in Disease Risk from Locations Observed with Error

Spatial cross-validation is not the right way to evaluate map accuracy

Characterizing Location and Classification Error Patterns in Time-Series Thematic Maps

Assessing the performance of spatial cross-validation approaches for models of spatially structured data

Factors Affecting Spatial Variation of Classification Uncertainty in an Image Object-based Vegetation Mapping

Spatial Autocorrelation among Automated Geocoding Errors and Its Effects on Testing for Disease Clustering

Spatial autoregressive model with measurement error in covariates

Evaluating the accuracy of binary classifiers for geomorphic applications

Prediction and model evaluation for space-time data

Logistic Regression with Misclassification in Binary Outcome Variables: a Method and Software

Thematic Classification Accuracy Assessment with Inherently Uncertain Boundaries: An Argument for Center-Weighted Accuracy Assessment Metrics

Accuracy Assessments and Uncertainty Analysis of Spatially Explicit Modeling for Land Use/cover Change and Urbanization: A Case in Beijing Metropolitan Area

Bayesian Hierarchical Spatial Regression Models for Spatial Data in the Presence of Missing Covariates with Applications