Abstract:Summary Models for predicting the distribution of organisms from environmental data are widespread in ecology and conservation biology. Their performance is invariably evaluated from the percentage success at predicting occurrence at test locations. Using logistic regression with real data from 34 families of aquatic invertebrates in 180 Himalayan streams, we illustrate how this widespread measure of predictive accuracy is affected systematically by the prevalence (i.e. the frequency of occurrence) of the target organism. Many evaluations of presence–absence models by ecologists are inherently misleading. With the same invertebrate models, we examined alternative performance measures used in remote sensing and medical diagnostics. We particularly explored receiver‐operating characteristic (ROC) plots, from which were derived (i) the area under each curve (AUC), considered an effective indicator of model performance independent of the threshold probability at which the presence of the target organism is accepted, and (ii) optimized probability thresholds that maximize the percentage of true absences and presences that are correctly identified. We also evaluated Cohen's kappa, a measure of the proportion of all possible cases of presence or absence that are predicted correctly after accounting for chance effects. AUC measures from ROC plots were independent of prevalence, but highly significantly correlated with the much more easily computed kappa. Moreover, when applied in predictive mode to test data, models with thresholds optimized by ROC erroneously overestimated true occurrence among scarcer organisms, often those of greatest conservation interest. We advocate caution in using ROC methods to optimize thresholds required for real prediction. Our strongest recommendation is that ecologists reduce their reliance on prediction success as a performance measure in presence–absence modelling. Cohen's kappa provides a simple, effective, standardized and appropriate statistic for evaluating or comparing presence–absence models, even those based on different statistical algorithms. None of the performance measures we examined tests the statistical significance of predictive accuracy, and we identify this as a priority area for research and development.

Accurately estimating correlations between demographic parameters: A comment on Deane et al. (2023)

Underestimation of Pearson’s product moment correlation statistic

Subtle methodological variations substantially impact correlation test results in ecological time series

A hierarchical model for jointly assessing ecological and anthropogenic impacts on animal demography

Evaluating presence-absence models in ecology: the need to account for prevalence

Reference-Invariant Inverse Covariance Estimation with Application to Microbial Network Recovery

Is non-informative Bayesian analysis appropriate for wildlife management: survival of San Joaquin Kit Fox and declines in amphibian populations

To remain modern the coexistence program requires modern statistical rigour

The difficulty of detecting inbreeding depression and its effect on conservation decisions

Abundance Versus Vital Rates: the Extent and Predictors of Inconsistent Conclusions on Avian Population Responses to Forest Loss and Degradation

Comparison of Inverse Wishart and Separation-Strategy Priors for Bayesian Estimation of Covariance Parameter Matrix in Growth Curve Analysis

Extending the DeLong algorithm for comparing areas under correlated receiver operating characteristic curves with missing data

High-dimensional Posterior Consistency in Multi-response Regression models with Non-informative Priors for Error Covariance Matrix

Robust Estimation of Polychoric Correlation

Intuitive joint priors for variance parameters

Valid auto-models for spatially autocorrelated occupancy and abundance data

Correlated Information Reduces Accuracy of Pioneering Decision-Makers

Improved state-space models for inference about spatial and temporal variation in abundance from count data

Reexamining the Relative Agreement Model of Opinion Dynamics.

Consequences of ignoring variable and spatially autocorrelated detection probability in spatial capture-recapture