Abstract:The methods for making statistical inferences in scientific analysis have diversified even within the frequentist branch of statistics, but comparison has been elusive. We approximate analytically and numerically the performance of Neyman-Pearson hypothesis testing, Fisher significance testing, information criteria, and evidential statistics (<a href="#B81">Royall, 1997</a>). This last approach is implemented in the form of evidence functions: statistics for comparing two models by estimating, based on data, their relative distance to the generating process (i.e., truth) (<a href="#B53">Lele, 2004</a>). A consequence of this definition is the salient property that the probabilities of misleading or weak evidence, error probabilities analogous to Type 1 and Type 2 errors in hypothesis testing, all approach 0 as sample size increases. Our comparison of these approaches focuses primarily on the frequency with which errors are made, both when models are correctly specified, and when they are misspecified, but also considers ease of interpretation. The error rates in evidential analysis all decrease to 0 as sample size increases even under model misspecification. Neyman-Pearson testing on the other hand, exhibits great difficulties under misspecification. The real Type 1 and Type 2 error rates can be less, equal to, or greater than the nominal rates depending on the nature of model misspecification. Under some reasonable circumstances, the probability of Type 1 error is an increasing function of sample size that can even approach 1! In contrast, under model misspecification an evidential analysis retains the desirable properties of always having a greater probability of selecting the best model over an inferior one and of having the probability of selecting the best model increase monotonically with sample size. We show that the evidence function concept fulfills the seeming objectives of model selection in ecology, both in a statistical as well as scientific sense, and that evidence functions are intuitive and easily grasped. We find that consistent information criteria are evidence functions but the MSE minimizing (or efficient) information criteria (e.g., AIC, AICc, TIC) are not. The error properties of the MSE minimizing criteria switch between those of evidence functions and those of Neyman-Pearson tests depending on models being compared.

Selective inference after likelihood- or test-based model selection in linear models

Exact post-selection inference for adjusted R squared selection

Selective Inference via Marginal Screening for High Dimensional Classification

Exact Post-Selection Inference for Sequential Regression Procedures

Bayesian variable selection in linear regression models with instrumental variables

Local asymptotics of selection models

Selective inference is easier with p-values

Non-asymptotic model selection for linear non least-squares estimation in regression models and inverse problems

Inference on Treatment Effects After Selection Amongst High-Dimensional Controls

Empirical-Likelihood-Based Inference for Partially Linear Models

A Nonprobabilistic Approach for Linear Model Selection

Evaluating the Relative Merits of Competing Models Based on Empirical Likelihood Ratio Test

Improved likelihood inference in generalized linear models

Subdata selection algorithm for linear model discrimination

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Parametric programming-based approximate selective inference for adaptive lasso, adaptive elastic net and group lasso

Errors in Statistical Inference Under Model Misspecification: Evidence, Hypothesis Testing, and AIC

Bayesian Robust Inference of Sample Selection Using Selection-t Models

More powerful post-selection inference, with application to the Lasso

Reconciling the Bayes Factor and Likelihood Ratio for Two Non-Nested Model Selection Problems

Selective Inference for Group-Sparse Linear Models