Abstract:This paper re-visits the problem of deciding between two simple hypotheses, the setting considered by Neyman and Pearson in developing their fundamental lemma. It studies the decision process induced by the most powerful test and the receiver operating characteristic function associated with this decision process. It addresses the question of how to report the decision arising from the decision function. It also examines the P-functional (the P-value statistic) and its role in the decision-making process. The impetus of this work is the continuing criticisms of statistical decision-making procedures that use the P-functional and a level of significance (LoS) of 0.05. A point made is that if one is going to use the value of the P-functional, then it should be used in an equivalent manner as the most powerful decision function, but if one wants to obtain from its value the degree of support for either hypotheses, then the value of its density under the alternative is the proper quantity to use. Replicability of results are discussed. Knowledge updating through Bayes theorem when given the decision or the value of the P-functional is also discussed, and it is argued that sequential learning is a coherent way of finding the truth. But the impact of publication bias is also demonstrated to be quite serious in the search for truth. It is argued that decision-makers are free to choose their own LoS, since the additional summary measures will automatically take their LoS choices into consideration. Three approaches for choosing an optimal LoS are discussed and a procedure for sample size determination is described. Ideas are illustrated by concrete problems and by the lady tea-tasting experiment of Fisher which ushered null hypothesis significance testing. It is hoped that by considering this fundamental setting of simple hypotheses, a better understanding of more complex settings will ensue.

Many perspectives on Deborah Mayo's "Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars"

Connections between statistical practice in elementary particle physics and the severity concept as discussed in Mayo's Statistical Inference as Severe Testing

Error and inference: an outsider stand on a frequentist philosophy

Discussion: On Arguments Concerning Statistical Principles

Comment on "The statistics wars and intellectual conflicts of interest" by D. Mayo

Perspective from the Literature on the Role of Expert Judgment in Scientific and Statistical Research and Practice

Statistical Inference: The Big Picture

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

Possibility-theoretic statistical inference offers performance and probativeness assurances

Semantic and Cognitive Tools to Aid Statistical Science: Replace Confidence and Significance by Compatibility and Surprise

Philosophy and the practice of Bayesian statistics

Testing Theories with Big Data : A SuperPower Approach

Connecting Simple and Precise P-values to Complex and Ambiguous Realities

Introduction to Statistical Modelling and Inference

Statistical Inference, Learning and Models in Big Data

Algorithmic and Statistical Perspectives on Large-Scale Data Analysis

Understanding Statistical Hypothesis Testing: The Logic of Statistical Inference

The Search for Truth through Data: NP Decision Processes, ROC Functions, $P$-Functionals, Knowledge Updating and Sequential Learning

Bayesian Statistical Pragmatism

Practical Bayesian Inference in Neuroscience: Or How I Learned To Stop Worrying and Embrace the Distribution

Bayesian Data Analysis: A Fresh Approach to Power Issues and Null Hypothesis Interpretation