Abstract:Bayesian cross-validation (CV) is a popular method for predictive model assessment that is simple to implement and broadly applicable. A wide range of CV schemes is available for time series applications, including generic leave-one-out (LOO) and K-fold methods, as well as specialized approaches intended to deal with serial dependence such as leave-future-out (LFO), h-block, and hv-block. Existing large-sample results show that both specialized and generic methods are applicable to models of serially-dependent data. However, large sample consistency results overlook the impact of sampling variability on accuracy in finite samples. Moreover, the accuracy of a CV scheme depends on many aspects of the procedure. We show that poor design choices can lead to elevated rates of adverse selection. In this paper, we consider the problem of identifying the regression component of an important class of models of data with serial dependence, autoregressions of order p with q exogenous regressors (ARX(p,q)), under the logarithmic scoring rule. We show that when serial dependence is present, scores computed using the joint (multivariate) density have lower variance and better model selection accuracy than the popular pointwise estimator. In addition, we present a detailed case study of the special case of ARX models with fixed autoregressive structure and variance. For this class, we derive the finite-sample distribution of the CV estimators and the model selection statistic. We conclude with recommendations for practitioners.

Cross validation for uncertain autoregressive model

Selection of Uncertain Differential Equations Using Cross Validation

Regularization and variable selection for infinite variance autoregressive models

Uncertain regression model with autoregressive time series errors

Variable selection in uncertain regression analysis with imprecise observations

Uncertain quantile autoregressive model

A note on the validity of cross-validation for evaluating autoregressive time series prediction

Prediction method of autoregressive moving average models for uncertain time series

Testing for variance changes in autoregressive models with unknown order

Ridge estimation for uncertain autoregressive moving average model with imprecise observations

Iterative Approximate Cross-Validation

Optimal model averaging based on forward-validation

Approximate leave-future-out cross-validation for Bayesian time series models

Overview of model validation for survival regression model with competing risks using melanoma study data

Ridge estimation of uncertain vector autoregressive model with imprecise data

Least absolute deviations for uncertain multivariate regression model

Cross-validatory model selection for Bayesian autoregressions with exogenous regressors

Variable selection for uncertain regression models based on elastic net method

Bootstrapping the Cross-Validation Estimate

Financial Volatility Forecasting with Range-Based Autoregressive Volatility Model

Approximate Cross-validation: Guarantees for Model Assessment and Selection