Abstract:Specifying, assessing, and selecting amongst candidate statistical models is fundamental to ecological research. Commonly used approaches to model selection are based on predictive scores and include information criteria such as Akaike's Information Criterion, and cross validation. Based on data splitting, cross validation is particularly versatile because it can be used even when it is not possible to derive a likelihood (e.g., many forms of machine learning) or count parameters precisely (e.g., mixed‐effects models). However, much of the literature on cross validation is technical and spread across statistical journals, making it difficult for ecological analysts to assess and choose amongst the wide range of options. Here we provide a comprehensive, accessible review that explains important‐‐but often overlooked‐‐technical aspects of cross validation for model selection, such as: bias correction, estimation uncertainty, choice of scores, and selection rules to mitigate overfitting. We synthesise the relevant statistical advances to make recommendations for the choice of cross‐validation technique and we present two ecological case studies to illustrate their application. In most instances, we recommend using exact or approximate leave‐one‐out cross validation to minimise bias, or otherwise k‐fold with bias correction if k<10. To mitigate overfitting when using cross validation, we recommend calibrated selection via our recently introduced modified one‐standard‐error rule. We advocate for the use of predictive scores in model selection across as range of typical modelling goals, such as exploration, hypothesis testing, and prediction, provided that models are specified in accordance with the stated goal. We also emphasise, as others have done, that inference on parameter estimates is biased if preceded by model selection and instead requires a carefully specified single model or further technical adjustments.

Selection of Uncertain Differential Equations Using Cross Validation

Cross validation for uncertain autoregressive model

Model Order Selection In Errors-In-Variables System Identification

Residual analysis and parameter estimation of uncertain differential equations

Is K-fold cross validation the best model selection method for Machine Learning?

Approximate Cross-validation: Guarantees for Model Assessment and Selection

Iterative Approximate Cross-Validation

Least squares parameter estimation for uncertain fractional differential equations and application to stock model

Variable Selection in Credit Risk Models for Chinese Listed Companies

Variable selection in uncertain regression analysis with imprecise observations

Portfolio Optimization Under the Uncertain Financial Model

Variable selection for uncertain regression models based on elastic net method

Model Selection Via Multifold Cross Validation

A survey of cross-validation procedures for model selection

Uncertain Hypothesis Test for Uncertain Differential Equations

Cross validation for model selection: a review with examples from ecology

A framework for the cross‐validation of categorical geostatistical simulations

A note on the validity of cross-validation for evaluating autoregressive time series prediction

Model Selection for Ordinary Differential Equations: A Statistical Testing Approach

Cross‐estimation for Decision Selection

UNCERTAINTY PORTFOLIO MODEL IN CROSS CURRENCY MARKETS