Abstract:The paper makes a case that the current discussions on replicability and the abuse of significance testing have overlooked a more general contributor to the untrustworthiness of published empirical evidence, which is the uninformed and recipe-like implementation of statistical modeling and inference. It is argued that this contributes to the untrustworthiness problem in several different ways, including [a] statistical misspecification, [b] unwarranted evidential interpretations of frequentist inference results, and [c] questionable modeling strategies that rely on curve-fitting. What is more, the alternative proposals to replace or modify frequentist testing, including [i] replacing p-values with observed confidence intervals and effects sizes, and [ii] redefining statistical significance, will not address the untrustworthiness of evidence problem since they are equally vulnerable to [a]–[c]. The paper calls for distinguishing between unduly data-dependant 'statistical results', such as a point estimate, a p-value, and accept/reject H0, from 'evidence for or against inferential claims'. The post-data severity (SEV) evaluation of the accept/reject H0 results, converts them into evidence for or against germane inferential claims. These claims can be used to address/elucidate several foundational issues, including (i) statistical vs. substantive significance, (ii) the large n problem, and (iii) the replicability of evidence. Also, the SEV perspective sheds light on the impertinence of the proposed alternatives [i]–[iii], and oppugns [iii] the alleged arbitrariness of framing H0 and H1 which is often exploited to undermine the credibility of frequentist testing.

Evidential Value in ANOVA Results in Favor of Fabrication

Evidential Value in ANOVA-Regression Results in Scientific Integrity Studies

Fraud detection with statistics: A comment on "Evidential Value in ANOVA-Regression Results in Scientific Integrity Studies" (Klaassen, 2015)

Statistical Detection of Potentially Fabricated Data

Statistics as a Social Activity: Attitudes toward Amalgamating Evidence

Evidential Analysis: An Alternative to Hypothesis Testing in Normal Linear Models

EIGENVALUE DISTRIBUTIONS OF VARIANCE COMPONENTS ESTIMATORS IN HIGH-DIMENSIONAL RANDOM EFFECTS MODELS

How the Post-Data Severity Converts Testing Results into Evidence for or against Pertinent Inferential Claims

Greenhouse–Geisser Adjustment and the ANOVA-Type Statistic: Cousins or Twins?

Data aggregation can lead to biased inferences in Bayesian linear mixed models and Bayesian analysis of variance.

Can We Mathematically Spot Possible Manipulation of Results in Research Manuscripts Using Benford's Law?

Repeated measures ANOVA and adjusted F-tests when sphericity is violated: which procedure is best?

The Influence of Validation Data on Logical and Scientific Interpretations of Forensic Expert Opinions

The Measurement of Statistical Evidence as the Basis for Statistical Reasoning

Bayesian Repeated-Measures Analysis of Variance: An Updated Methodology Implemented in JASP

Beyond ANOVA: An Introduction to Structural Equation Models for Experimental Designs

Conflict diagnostics for evidence synthesis in a multiple testing framework

Too Good to Be True? Understanding Change in Organizational Outcomes

The Concept of Statistical Evidence: Historical Roots and Current Developments

Demonstrative Evidence and the Use of Algorithms in Jury Trials

Permutation Strategies for Inference in ANOVA‐Based Models for Nonorthogonal Designs Including Continuous Covariates