Abstract:This paper re-visits the problem of deciding between two simple hypotheses, the setting considered by Neyman and Pearson in developing their fundamental lemma. It studies the decision process induced by the most powerful test and the receiver operating characteristic function associated with this decision process. It addresses the question of how to report the decision arising from the decision function. It also examines the P-functional (the P-value statistic) and its role in the decision-making process. The impetus of this work is the continuing criticisms of statistical decision-making procedures that use the P-functional and a level of significance (LoS) of 0.05. A point made is that if one is going to use the value of the P-functional, then it should be used in an equivalent manner as the most powerful decision function, but if one wants to obtain from its value the degree of support for either hypotheses, then the value of its density under the alternative is the proper quantity to use. Replicability of results are discussed. Knowledge updating through Bayes theorem when given the decision or the value of the P-functional is also discussed, and it is argued that sequential learning is a coherent way of finding the truth. But the impact of publication bias is also demonstrated to be quite serious in the search for truth. It is argued that decision-makers are free to choose their own LoS, since the additional summary measures will automatically take their LoS choices into consideration. Three approaches for choosing an optimal LoS are discussed and a procedure for sample size determination is described. Ideas are illustrated by concrete problems and by the lady tea-tasting experiment of Fisher which ushered null hypothesis significance testing. It is hoped that by considering this fundamental setting of simple hypotheses, a better understanding of more complex settings will ensue.

Neyman-Pearson Hypothesis Testing, Epistemic Reliability and Pragmatic Value-Laden Asymmetric Error Risks

Beyond Neyman-Pearson: E-values enable hypothesis testing with a data-driven alpha

The Lack of Robustness of a Statistic Based on the Neyman-Pearson Lemma to Violations of Its Underlying Assumptions

Connecting Simple and Precise P-values to Complex and Ambiguous Realities

Hypothesis Testing in Econometrics

On Some Assumptions of the Null Hypothesis Statistical Testing

Confidence distributions and hypothesis testing

Goodness of fit by Neyman-Pearson testing

The Search for Truth through Data: NP Decision Processes, ROC Functions, $P$-Functionals, Knowledge Updating and Sequential Learning

Understanding the implications of low knowledge and high uncertainty in risk studies

Towards a theory for testing statistical hypothesis: Multivariate mean with nuisance covariance matrix

Non-Stochastic Hypothesis Testing with Application to Privacy Against Hypothesis-Testing Adversary

Kernel Robust Hypothesis Testing

Safe Testing

A Likelihood-based Alternative to Null Hypothesis Significance Testing

Valid p-Values and Expectations of p-Values Revisited

Post-hoc Hypothesis Testing

Interpretable hypothesis tests

Significance level and positivity bias as causes for high rate of non-reproducible scientific results?

A Practitioner's Guide to Multiple Testing Error Rates

Testing in the Presence of Nuisance Parameters: Some Comments on Tests Post-Model-Selection and Random Critical Values