Abstract:<p>While the application of machine-learning algorithms has been highly simplified in the last years due to their well-documented integration in commonly used statistical programming languages (such as R or Python), there are several practical challenges in the field of ecological modeling related to unbiased performance estimation. One is the influence of spatial autocorrelation in both hyperparameter tuning and performance estimation. Grouped cross-validation strategies have been proposed in recent years in environmental as well as medical contexts to reduce bias in predictive performance. In this study we show the effects of spatial autocorrelation on hyperparameter tuning and performance estimation by comparing several widely used machine-learning algorithms such as boosted regression trees (BRT), k-nearest neighbor (KNN), random forest (RF) and support vector machine (SVM) with traditional parametric algorithms such as logistic regression (GLM) and semi-parametric ones like generalized additive models (GAM) in terms of predictive performance. Spatial and non-spatial cross-validation methods were used to evaluate model performances aiming to obtain bias-reduced performance estimates. A detailed analysis on the sensitivity of hyperparameter tuning when using different resampling methods (spatial/non-spatial) was performed. As a case study the spatial distribution of forest disease (<em>Diplodia sapinea</em>) in the Basque Country (Spain) was investigated using common environmental variables such as temperature, precipitation, soil and lithology as predictors. Random Forest (mean Brier score estimate of 0.166) outperformed all other methods with regard to predictive accuracy. Though the sensitivity to hyperparameter tuning differed between the ML algorithms, there were in most cases no substantial differences between spatial and non-spatial partitioning for hyperparameter tuning. However, spatial hyperparameter tuning maintains consistency with spatial estimation of classifier performance and should be favored over non-spatial hyperparameter optimization. High performance differences (up to 47%) between the bias-reduced (spatial cross-validation) and overoptimistic (non-spatial cross-validation) cross-validation settings showed the high need to account for the influence of spatial autocorrelation. Overoptimistic performance estimates may lead to false actions in ecological decision making based on biased model predictions.</p>

Exploring the impact of spatial autocorrelation on optimistic bias in cross-validation and assessing the effectiveness of spatial cross-validation

Estimating the prediction performance of spatial models via spatial k-fold cross validation

Spatial cross-validation is not the right way to evaluate map accuracy

Accounting for Spatial Autocorrelation in Algorithm-Driven Hedonic Models: A Spatial Cross-Validation Approach

Assessing the performance of spatial cross-validation approaches for models of spatially structured data

On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction

Hyperparameter tuning and performance assessment of statistical and machine-learning algorithms using spatial data

Consistent Validation for Predictive Methods in Spatial Settings

Importance of spatial predictor variable selection in machine learning applications -- Moving from data reproduction to spatial prediction

Demystifying Spatial Confounding

Spatial Autocorrelation among Automated Geocoding Errors and Its Effects on Testing for Disease Clustering

Effects of Intrinsic Sources of Spatial Autocorrelation on Spatial Regression Modelling

Spatial+: a novel approach to spatial confounding

Risk and cross validation in ridge regression with correlated samples

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data

A note on the validity of cross-validation for evaluating autoregressive time series prediction

Spatial Autocorrelation Approaches to Testing Residuals from Least Squares Regression.

Spatial prediction variance estimation based on covariance penalty

Robust Variable Selection with Exponential Squared Loss for the Spatial Autoregressive Model

Testing for Spatial Autocorrelation among the Residuals of the Geographically Weighted Regression

Accounting for spatial correlation in the scan statistic