Abstract:<p>While the application of machine-learning algorithms has been highly simplified in the last years due to their well-documented integration in commonly used statistical programming languages (such as R or Python), there are several practical challenges in the field of ecological modeling related to unbiased performance estimation. One is the influence of spatial autocorrelation in both hyperparameter tuning and performance estimation. Grouped cross-validation strategies have been proposed in recent years in environmental as well as medical contexts to reduce bias in predictive performance. In this study we show the effects of spatial autocorrelation on hyperparameter tuning and performance estimation by comparing several widely used machine-learning algorithms such as boosted regression trees (BRT), k-nearest neighbor (KNN), random forest (RF) and support vector machine (SVM) with traditional parametric algorithms such as logistic regression (GLM) and semi-parametric ones like generalized additive models (GAM) in terms of predictive performance. Spatial and non-spatial cross-validation methods were used to evaluate model performances aiming to obtain bias-reduced performance estimates. A detailed analysis on the sensitivity of hyperparameter tuning when using different resampling methods (spatial/non-spatial) was performed. As a case study the spatial distribution of forest disease (<em>Diplodia sapinea</em>) in the Basque Country (Spain) was investigated using common environmental variables such as temperature, precipitation, soil and lithology as predictors. Random Forest (mean Brier score estimate of 0.166) outperformed all other methods with regard to predictive accuracy. Though the sensitivity to hyperparameter tuning differed between the ML algorithms, there were in most cases no substantial differences between spatial and non-spatial partitioning for hyperparameter tuning. However, spatial hyperparameter tuning maintains consistency with spatial estimation of classifier performance and should be favored over non-spatial hyperparameter optimization. High performance differences (up to 47%) between the bias-reduced (spatial cross-validation) and overoptimistic (non-spatial cross-validation) cross-validation settings showed the high need to account for the influence of spatial autocorrelation. Overoptimistic performance estimates may lead to false actions in ecological decision making based on biased model predictions.</p>

A Monte Carlo resampling framework for implementing goodness‐of‐fit tests in spatial capture‐recapture models

Modelling spatially autocorrelated detection probabilities in spatial capture-recapture using random effects

Geostatistical capture–recapture models

A flexible and efficient Bayesian implementation of point process models for spatial capture‐recapture data

A formal goodness-of-fit test for spatial binary Markov random field models

A local evaluation of the individual state-space to scale up Bayesian spatial capture-recapture

Consequences of ignoring variable and spatially autocorrelated detection probability in spatial capture-recapture

Integrating Resource Selection Information with Spatial Capture-Recapture

Characteristic Function-Based Testing for Multifactor Continuous-Time Markov Models Via Nonparametric Regression

Incorporating Memory into Continuous-Time Spatial Capture-Recapture Models

Bayesian Model Selection for a Class of Spatially-Explicit Capture Recapture Models

Using lineups to evaluate goodness of fit of animal movement models

A sequential Monte Carlo algorithm for data assimilation problems in ecology

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data

A Goodness-of-fit Test for Parametric and Semi-Parametric Models in Multiresponse Regression

Hyperparameter tuning and performance assessment of statistical and machine-learning algorithms using spatial data

Goodness-of-fit testing in high-dimensional generalized linear models

Explicit modeling of density dependence in spatial capture-recapture models

A flexible framework for spatial capture-recapture with unknown identities

A Goodness-of-Fit Test for Statistical Models

A Unifying Model for Capture-Recapture and Distance Sampling Surveys of Wildlife Populations