Abstract:The methods for making statistical inferences in scientific analysis have diversified even within the frequentist branch of statistics, but comparison has been elusive. We approximate analytically and numerically the performance of Neyman-Pearson hypothesis testing, Fisher significance testing, information criteria, and evidential statistics (<a href="#B81">Royall, 1997</a>). This last approach is implemented in the form of evidence functions: statistics for comparing two models by estimating, based on data, their relative distance to the generating process (i.e., truth) (<a href="#B53">Lele, 2004</a>). A consequence of this definition is the salient property that the probabilities of misleading or weak evidence, error probabilities analogous to Type 1 and Type 2 errors in hypothesis testing, all approach 0 as sample size increases. Our comparison of these approaches focuses primarily on the frequency with which errors are made, both when models are correctly specified, and when they are misspecified, but also considers ease of interpretation. The error rates in evidential analysis all decrease to 0 as sample size increases even under model misspecification. Neyman-Pearson testing on the other hand, exhibits great difficulties under misspecification. The real Type 1 and Type 2 error rates can be less, equal to, or greater than the nominal rates depending on the nature of model misspecification. Under some reasonable circumstances, the probability of Type 1 error is an increasing function of sample size that can even approach 1! In contrast, under model misspecification an evidential analysis retains the desirable properties of always having a greater probability of selecting the best model over an inferior one and of having the probability of selecting the best model increase monotonically with sample size. We show that the evidence function concept fulfills the seeming objectives of model selection in ecology, both in a statistical as well as scientific sense, and that evidence functions are intuitive and easily grasped. We find that consistent information criteria are evidence functions but the MSE minimizing (or efficient) information criteria (e.g., AIC, AICc, TIC) are not. The error properties of the MSE minimizing criteria switch between those of evidence functions and those of Neyman-Pearson tests depending on models being compared.

Mean Square Error Matrix Properties of Bayes Estimation for Incorrect Prior Information under Misspecification

The Robustness of Bayes Linear Unbiased Estimations under Misspecified Prior Assumption

A Bound on Mean-Square Estimation Error with Background Parameter Mismatch

The Superiority of Bayes Linear Unbiased Minimum Variance Estimator with Respect to Ridge Estimator

Moments and Root-Mean-Square Error of the Bayesian MMSE Estimator of Classification Error in the Gaussian Model

On a Biased Prediction Based on Optimal Mean Square Error Criterion

Small Sample Properties for Mixed Estimator of Regression Coefficients under Misspecified Regression Model

Correcting the estimator for the mean vectors in a multivariate errors-in-variables regression model

A Bilateral Bound on the Mean-Square Error for Estimation in Model Mismatch

Superiority of Empirical Bayes Estimation of Error Variance in Linear Model

Superiorities of Simultaneous Bayes Estimation of Regression Coefficients and Error-Variance in Linear Model

On estimation of mean squared errors of benchmarked empirical Bayes estimators

Empirical Bayes in Bayesian learning: understanding a common practice

The Superiorities of Bayes Linear Minimum Risk Estimation in Linear Model

Mean Square Prediction Error of Misspecified Gaussian Process Models

Errors in Statistical Inference Under Model Misspecification: Evidence, Hypothesis Testing, and AIC

Low-rank matrix estimation in multi-response regression with measurement errors: Statistical and computational guarantees.

Bayes estimator for the inequality-constrained regression model*

Bayesian inference using least median of squares and least trimmed squares in models with independent or correlated errors and outliers

Misspecified Exponential Regressions: Estimation, Interpretation, and Average Marginal Effects

Single-index relative error regression models