Abstract:Information criteria provide a cogent approach for identifying models that provide an optimal balance between the competing objectives of goodness-of-fit and parsimony. Models that better conform to a dataset are often more complex, yet such models are plagued by greater variability in estimation and prediction. Conversely, overly simplistic models reduce variability at the cost of increases in bias. Asymptotically efficient criteria are those that, for large samples, select the fitted candidate model whose predictors minimize the mean squared prediction error, optimizing between prediction bias and variability. In the context of prediction, asymptotically efficient criteria are thus a preferred tool for model selection, with the Akaike information criterion (AIC) being among the most widely used. However, asymptotic efficiency relies upon the assumption of a panel of validation data generated independently from, but identically to, the set of training data. We argue that assuming identically distributed training and validation data is misaligned with the premise of prediction and often violated in practice. This is most apparent in a regression context, where assuming training/validation data homogeneity requires identical panels of regressors. We therefore develop a new class of predictive information criteria (PIC) that do not assume training/validation data homogeneity and are shown to generalize AIC to the more practically relevant setting of training/validation data heterogeneity. The analytic properties and predictive performance of these new criteria are explored within the traditional regression framework. We consider both simulated and real-data settings. Software for implementing these methods is provided in the R package, picR , available through CRAN.

Deviance Information Criterion for Model Selection : Justification and Variation ∗

Deviance Information Criterion for Bayesian Model Selection: Justification and Variation

Deviance Information Criterion for Latent Variable Models and Misspecified Models

Integrated Deviance Information Criterion for Latent Variable Models

Deviance Information Criterion for Comparing VAR Models

Integrated deviance information criterion for spatial autoregressive models with heteroskedasticity

A simple application of FIC to model selection

Gibbs-Based Information Criteria and the Over-Parameterized Regime

Variable Selection Using a Smooth Information Criterion for Distributional Regression Models

Decision Curve Analysis: a Technical Note

Posterior Averaging Information Criterion

A Simple Information Criterion for Variable Selection in High‐Dimensional Regression

A new class of information criteria for improved prediction in the presence of training/validation data heterogeneity

Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC).

Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood

On the Equivalence of Factorized Information Criterion Regularization and the Chinese Restaurant Process Prior

Fit for a Bayesian: An Evaluation of PPP and DIC for Structural Equation Modeling

Kullback-Leibler Divergence and Akaike Information Criterion in General Hidden Markov Models

PanIC: consistent information criteria for general model selection problems

Consistent Model Selection Procedure for Random Coefficient INAR Models

A Theoretical Investigation of Several Model Selection Criteria for Dimensionality Reduction