Abstract:When comparing multiple models of species distribution, models yielding higher predictive performance are clearly to be favored. A more difficult question is how to decide whether even the best model is "good enough". Here, we clarify key choices and metrics related to evaluating the predictive performance of presence-absence models. We use a hierarchical case study to evaluate how four metrics of predictive performance (AUC, Tjur's R 2, max-Kappa, and max-TSS) relate to each other, the random and fixed effects parts of the model, the spatial scale at which predictive performance is measured, and the cross-validation strategy chosen. We demonstrate that the very same metric can achieve different values for the very same model, even when similar cross-validation strategies are followed, depending on the spatial scale at which predictive performance is measured. Among metrics, Tjur's R 2 and max-Kappa generally increase with species' prevalence, whereas AUC and max-TSS are largely independent of prevalence. Thus, Tjur's R 2 and max-Kappa often reach lower values when measured at the smallest scales considered in the study, while AUC and max-TSS reaching similar values across the different spatial levels included in the study. However, they provide complementary insights on predictive performance. The very same model may appear excellent or poor not only due to the applied metric, but also how predictive performance is exactly calculated, calling for great caution on the interpretation of predictive performance. The most comprehensive evaluation of predictive performance can be obtained by evaluating predictive performance through the combination of measures providing complementary insights. Instead of following simple rules of thumb or focusing on absolute values, we recommend comparing the achieved predictive performance to the researcher's own a priori expectations on how easy it is to make predictions related to the same question that the model is used for.

Testing differences in predictive ability: A tutorial

More accurate tests for the statistical significance of result differences

Predictive ability tests with possibly overlapping models

A Comparative Review of Methods for Comparing Means Using Partially Paired Data

Statistical Methods for Comparing Predictive Values in Medical Diagnosis

Mistaken identities lead to missed opportunities: Testing for mean differences in partially matched data

Comparison of approaches for local testing with functional test statistics

A Note on Comparison of F-measures

Model predictivity assessment: incremental test-set selection and accuracy evaluation

Tests for equal forecast accuracy under heteroskedasticity

Testing the Consistency of Performance Scores Reported for Binary Classification Problems

Correlation‐based tests of predictability

Testing for improvement in prediction model performance

Statistical hypothesis testing versus machine-learning binary classification: distinctions and guidelines

What is different between these datasets?

Evaluation metrics and statistical tests for machine learning

Evaluating the predictive performance of presence-absence models: Why can the same model appear excellent or poor?

Fairer and more accurate, but for whom?

Predicting Performance During Tutoring with Models of Recent Performance

Understanding Prediction Discrepancies in Machine Learning Classifiers

Bonferroni‐Type Tests for Return Predictability With Possibly Trending Predictors