Rejoinder to "Multiple Testing for Exploratory Research"

Jelle J. Goeman,Aldo Solari

DOI: https://doi.org/10.1214/11-STS356REJ

2012-08-16

Abstract:Rejoinder to "Multiple Testing for Exploratory Research" by J. J. Goeman, A. Solari [<a class="link-https" data-arxiv-id="1208.2841" href="https://arxiv.org/abs/1208.2841">arXiv:1208.2841</a>].

Methodology

What problem does this paper attempt to address?

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Xinyue Zhou,Joel R. Levin

DOI: https://doi.org/10.22237/jmasm/1099268100

2004-01-01

Journal of Modern Applied Statistical Methods

Abstract:This article extends Scheffé's modified (sequential) multiple-comparison procedure in one-way analysis-of-variance to other analysis situations, including interaction comparisons in factorial ANOVA designs, tests of partial regression coefficients in multiple-regression analysis, and comparisons of means in one-factor multivariate analyses of variance. Researchers who are concerned with maintaining familywise Type I error rates while increasing statistical power relative to the original (simultaneous) Scheffé-based procedures are encouraged to consider these improved multiple-comparison methods. Copyright © 2004 JMASM, Inc.
Testing Hypotheses When You Have More Than a Few

Peter A. Rogerson

DOI: https://doi.org/10.1111/gean.12412

IF: 3.566

2024-08-31

Geographical Analysis

Abstract:A common issue faced by spatial analysts is that of multiple testing. When hypotheses are tested at multiple points in time or space, care must often be taken to avoid results containing too many false positives. There are many ways to address this outcome, and these are reviewed in this article. We begin with a review of some of the basic, longstanding approaches to multiple testing. This is followed by a summary of the more recent objective of controlling the false discovery rate and the effects of spatial autocorrelation on it. The number of true null hypotheses is an important quantity, and some approaches to its estimation are reviewed. In the literature on spatial analysis, there have been several newer approaches to multiple testing, and these are also reviewed. These include some recent methods outside of the literature in geography, but they have potential applicability for many of the problems addressed by geographers, especially since they focus upon the discovery of clusters. The article includes an illustration and closes with some ideas for taking further steps in treating multiple hypotheses in the context of methods commonly used in geographical analysis.

geography
Rejoinder to Article by Berger, Bernardo, and Sun

James O. Berger,Jose M. Bernardo,Dongchu Sun

DOI: https://doi.org/10.1214/15-BA943

2015-04-27

Abstract:Rejoinder to Overall Objective Priors by James O. Berger, Jose M. Bernardo, Dongchu Sun [<a class="link-https" data-arxiv-id="1504.02689" href="https://arxiv.org/abs/1504.02689">arXiv:1504.02689</a>]

Statistics Theory
MultipleTesting.com: a tool for life science researchers for multiple hypothesis testing correction

Otília Menyhárt,Boglárka Weltz,Balázs Győrffy

DOI: https://doi.org/10.1101/2021.01.11.426197

2021-01-11

Abstract:ABSTRACT Scientists from nearly all disciplines face the problem of simultaneously evaluating many hypotheses. Conducting multiple comparisons increases the likelihood that a non-negligible proportion of associations will be false positives, clouding real discoveries. Drawing valid conclusions require taking into account the number of performed statistical tests and adjusting the statistical confidence measures. Several strategies exist to overcome the problem of multiple hypothesis testing. We aim to summarize critical statistical concepts and widely used correction approaches while also draw attention to frequently misinterpreted notions of statistical inference. We provide a step-by-step description of each multiple-testing correction method with clear examples and present an easy-to-follow guide for selecting the most suitable correction technique. To facilitate multiple-testing corrections, we developed a fully automated solution not requiring programming skills or the use of a command line. Our registration free online tool is available at www.multipletesting.com and compiles the five most frequently used adjustment tools, including the Bonferroni, the Holm (step-down), the Hochberg (step-up) corrections, allows to calculate False Discovery Rates (FDR) and q-values. The current summary provides a much needed practical synthesis of basic statistical concepts regarding multiple hypothesis testing in a comprehensible language with well-illustrated examples. The web tool will fill the gap for life science researchers by providing a user-friendly substitute for command-line alternatives.
Dealing with multiple testing: To adjust or not to adjust

Yudi Pawitan,Arvid Sjölander

DOI: https://doi.org/10.48550/arXiv.2010.02205

2020-10-04

Other Statistics

Abstract:Multiple testing problems arise naturally in scientific studies because of the need to capture or convey more information with more variables. The literature is enormous, but the emphasis is primarily methodological, providing numerous methods with their mathematical justification and practical implementation. Our aim is to highlight the logical issues involved in the application of multiple testing adjustment.
Multiple testing in statistical analysis of systems-based information retrieval experiments

Benjamin A. Carterette

DOI: https://doi.org/10.1145/2094072.2094076

IF: 4.657

2012-02-01

ACM Transactions on Information Systems

Abstract:High-quality reusable test collections and formal statistical hypothesis testing together support a rigorous experimental environment for information retrieval research. But as Armstrong et al. [2009b] recently argued, global analysis of experiments suggests that there has actually been little real improvement in ad hoc retrieval effectiveness over time. We investigate this phenomenon in the context of simultaneous testing of many hypotheses using a fixed set of data. We argue that the most common approaches to significance testing ignore a great deal of information about the world. Taking into account even a fairly small amount of this information can lead to very different conclusions about systems than those that have appeared in published literature. We demonstrate how to model a set of IR experiments for analysis both mathematically and practically, and show that doing so can cause p -values from statistical hypothesis tests to increase by orders of magnitude. This has major consequences on the interpretation of experimental results using reusable test collections: it is very difficult to conclude that anything is significant once we have modeled many of the sources of randomness in experimental design and analysis.

computer science, information systems
More Powerful Multiple Testing in Randomized Experiments with Non-Compliance

Joseph J. Lee,Laura Forastiere,Luke Miratrix,Natesh S. Pillai

DOI: https://doi.org/10.48550/arXiv.1605.07242

2016-05-24

Abstract:Two common concerns raised in analyses of randomized experiments are (i) appropriately handling issues of non-compliance, and (ii) appropriately adjusting for multiple tests (e.g., on multiple outcomes or subgroups). Although simple intention-to-treat (ITT) and Bonferroni methods are valid in terms of type I error, they can each lead to a substantial loss of power; when employing both simultaneously, the total loss may be severe. Alternatives exist to address each concern. Here we propose an analysis method for experiments involving both features that merges posterior predictive $p$-values for complier causal effects with randomization-based multiple comparisons adjustments; the results are valid familywise tests that are doubly advantageous: more powerful than both those based on standard ITT statistics and those using traditional multiple comparison adjustments. The operating characteristics and advantages of our method are demonstrated through a series of simulated experiments and an analysis of the United States Job Training Partnership Act (JTPA) Study, where our methods lead to different conclusions regarding the significance of estimated JTPA effects.

Methodology,Applications
A model of multiple hypothesis testing

Davide Viviano,Kaspar Wuthrich,Paul Niehaus

2024-04-26

Abstract:Multiple hypothesis testing practices vary widely, without consensus on which are appropriate when. This paper provides an economic foundation for these practices designed to capture processes of scientific communication, such as regulatory approval on the basis of clinical trials. In studies of multiple treatments or sub-populations, adjustments may be appropriate depending on scale economies in the research production function, with control of classical notions of compound errors emerging in some but not all cases. In studies with multiple outcomes, indexing is appropriate and adjustments to test levels may be appropriate if the intended audience is heterogeneous. Data on actual costs in the drug approval process suggest both that some adjustment is warranted in that setting and that standard procedures are overly conservative.

General Economics,Econometrics
A Practitioner's Guide to Multiple Testing Error Rates

Jonathan Rosenblatt

DOI: https://doi.org/10.48550/arXiv.1304.4920

2013-06-25

Abstract:It is quite common in modern research, for a researcher to test many hypotheses. The statistical (frequentist) hypothesis testing framework, does not scale with the number of hypotheses in the sense that naively performing many hypothesis tests will probably yield many false findings. Indeed, statistical "significance" is evidence for the presence of a signal within the noise expected in a single test, not in a multitude. In order to protect himself from an uncontrolled number of erroneous findings, a researcher has to consider of the type or errors he wishes to avoid and select the adequate procedure for that particular error type and data structure. A quick search of the tag [multiple-comparisons] in the statistics Questions & Answers web site Cross Validates (<a class="link-external link-http" href="http://stats.stackexchange.com" rel="external noopener nofollow">this http URL</a>) demonstrates the amount of confusion this task can actually cause. This was also a point made at the 2009 Multiple Comparisons conference in Tokyo. In an attempt to offer guidance, we review possible error types for multiple testing, and demonstrate them with some practical examples, which clarify the formalism. Finally, we include some notes on the software implementations of the methods discussed. The emphasis of this manuscript is on the error-rates, and not on the procedures themselves. We do try to name several procedures in this manuscript where appropriate. P-value adjustment will not be discussed as it is procedure specific. I.e., it is the choice of a procedure that defines the p-value adjustment, and not the error rate itself. Simultaneous confidence intervals will, also, not be discussed.

Methodology
Rejoinder: Struggles with Survey Weighting and Regression Modeling

Andrew Gelman

DOI: https://doi.org/10.1214/088342307000000203

2007-10-26

Abstract:Rejoinder: Struggles with Survey Weighting and Regression Modeling [<a class="link-https" data-arxiv-id="0710.5005" href="https://arxiv.org/abs/0710.5005">arXiv:0710.5005</a>]

Methodology
On particle acceleration and trapping by Poynting flux dominated flows

Gunnar Paesold,Eric G. Blackman,Peter Messmer

DOI: https://doi.org/10.1088/0741-3335/47/11/005

2005-08-25

Abstract:Using particle-in-cell (PIC) simulations, we study the evolution of a strongly magnetized plasma slab propagating into a finite density ambient medium. Like previous work, we find that the slab breaks into discrete magnetic pulses. The subsequent evolution is consistent with diamagnetic relativistic pulse acceleration of \cite{liangetal2003}. Unlike previous work, we use the actual electron to proton mass ratio and focus on understanding trapping vs. transmission of the ambient plasma by the pulses and on the particle acceleration spectra. We find that the accelerated electron distribution internal to the slab develops a double-power law. We predict that emission from reflected/trapped external electrons will peak after that of the internal electrons. We also find that the thin discrete pulses trap ambient electrons but allow protons to pass through, resulting in less drag on the pulse than in the case of trapping of both species. Poynting flux dominated scenarios have been proposed as the driver of relativistic outflows and particle acceleration in the most powerful astrophysical jets.

Astrophysics,Plasma Physics
A Family of Bayes Multiple Testing Procedures

Arthur Cohen,H. B. Sackrowitz,Minya Xu,Steven Buyske

DOI: https://doi.org/10.1093/biomet/asn013

IF: 3.0279

2008-01-01

Biometrika

Abstract:Under the model of independent test statistics, we propose a two-parameter family of Bayes multiple testing procedures. The two parameters can be viewed as tuning parameters. Using the Benjamini - Hochberg step-up procedure for controlling false discovery rate as a baseline for conservativeness, we choose the tuning parameters to compromise between the operating characteristics of that procedure and a less conservative procedure that focuses on alternatives that a priori might be considered likely or meaningful. The Bayes procedures do not have the theoretical and practical shortcomings of the popular stepwise procedures. In terms of the number of mistakes, simulations for two examples indicate that over a large segment of the parameter space, the Bayes procedure is preferable to the step-up procedure. Another desirable feature of the procedures is that they are computationally feasible for any number of hypotheses.
What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research

Klaus E Meyer,Arjen van Witteloostuijn,Sjoerd Beugelsdijk

DOI: https://doi.org/10.1057/s41267-017-0078-8

IF: 11.6

2017-05-19

Journal of International Business Studies

Abstract:Social science research has recently been subject to considerable criticism regarding the validity and power of empirical tests published in leading journals, and business scholarship is no exception. Transparency and replicability of empirical findings are essential to build a cumulative body of scholarly knowledge. Yet current practices are under increased scrutiny to achieve these objectives. JIBS is therefore discussing and revising its editorial practices to enhance the validity of empirical research. In this editorial, we reflect on best practices with respect to conducting, reporting, and discussing the results of quantitative hypothesis-testing research, and we develop guidelines for authors to enhance the rigor of their empirical work. This will not only help readers to assess empirical evidence comprehensively, but also enable subsequent research to build a cumulative body of empirical knowledge.

management,business
The sequential rejection principle of familywise error control

Jelle J. Goeman,Aldo Solari

DOI: https://doi.org/10.48550/arXiv.1211.3313

2012-11-14

Statistics Theory

Abstract:Closed testing and partitioning are recognized as fundamental principles of familywise error control. In this paper, we argue that sequential rejection can be considered equally fundamental as a general principle of multiple testing. We present a general sequentially rejective multiple testing procedure and show that many well-known familywise error controlling methods can be constructed as special cases of this procedure, among which are the procedures of Holm, Shaffer and Hochberg, parallel and serial gatekeeping procedures, modern procedures for multiple testing in graphs, resampling-based multiple testing procedures and even the closed testing and partitioning procedures themselves. We also give a general proof that sequentially rejective multiple testing procedures strongly control the familywise error if they fulfill simple criteria of monotonicity of the critical values and a limited form of weak familywise error control in each single step. The sequential rejection principle gives a novel theoretical perspective on many well-known multiple testing procedures, emphasizing the sequential aspect. Its main practical usefulness is for the development of multiple testing procedures for null hypotheses, possibly logically related, that are structured in a graph. We illustrate this by presenting a uniform improvement of a recently published procedure.
A New Multiple Testing Method in the Dependent Case

Arthur Cohen,Harold B. Sackrowitz,Minya Xu

DOI: https://doi.org/10.1214/08-aos616

2009-01-01

Abstract:The most popular multiple testing procedures are stepwise procedures based on P-values for individual test statistics. Included among these are the false discovery rate (FDR) controlling procedures of Benjamini-Hochberg [J. Roy. Statist. Soc. Ser B 57 (1995) 289-300] and their offsprings. Even for models that entail dependent data, P-values based on marginal distributions are used. Unlike such methods, the new method takes dependency into account at all stages. Furthermore, the P-value procedures often lack an intuitive convexity property, which is needed for admissibility. Still further, the new methodology is computationally feasible. If the number of tests is large and the proportion of true alternatives is less than say 25 percent, simulations demonstrate a clear preference for the new methodology. Applications are detailed for models such as testing treatments against control (or any intraclass correlation model), testing for change points and testing means when correlation is successive.
Addressing researcher degrees of freedom through minP adjustment

Maximilian M. Mandl,Andrea S. Becker-Pennrich,Ludwig C. Hinske,Sabine Hoffmann,Anne-Laure Boulesteix

DOI: https://doi.org/10.1186/s12874-024-02279-2

2024-07-18

BMC Medical Research Methodology

Abstract:When different researchers study the same research question using the same dataset they may obtain different and potentially even conflicting results. This is because there is often substantial flexibility in researchers' analytical choices, an issue also referred to as "researcher degrees of freedom". Combined with selective reporting of the smallest p -value or largest effect, researcher degrees of freedom may lead to an increased rate of false positive and overoptimistic results. In this paper, we address this issue by formalizing the multiplicity of analysis strategies as a multiple testing problem. As the test statistics of different analysis strategies are usually highly dependent, a naive approach such as the Bonferroni correction is inappropriate because it leads to an unacceptable loss of power. Instead, we propose using the "minP" adjustment method, which takes potential test dependencies into account and approximates the underlying null distribution of the minimal p -value through a permutation-based procedure. This procedure is known to achieve more power than simpler approaches while ensuring a weak control of the family-wise error rate. We illustrate our approach for addressing researcher degrees of freedom by applying it to a study on the impact of perioperative on post-operative complications after neurosurgery. A total of 48 analysis strategies are considered and adjusted using the minP procedure. This approach allows to selectively report the result of the analysis strategy yielding the most convincing evidence, while controlling the type 1 error—and thus the risk of publishing false positive results that may not be replicable.

health care sciences & services
Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel,Barnabas Szaszi,Gustav Nilsonne,Olmo R van den Akker,Casper J Albers,Marcel ALM van Assen,Jojanneke A Bastiaansen,Daniel Benjamin,Udo Boehm,Rotem Botvinik-Nezer,Laura F Bringmann,Niko A Busch,Emmanuel Caruyer,Andrea M Cataldo,Nelson Cowan,Andrew Delios,Noah NN van Dongen,Chris Donkin,Johnny B van Doorn,Anna Dreber,Gilles Dutilh,Gary F Egan,Morton Ann Gernsbacher,Rink Hoekstra,Sabine Hoffmann,Felix Holzmeister,Juergen Huber,Magnus Johannesson,Kai J Jonas,Alexander T Kindel,Michael Kirchler,Yoram K Kunkels,D Stephen Lindsay,Jean-Francois Mangin,Dora Matzke,Marcus R Munafò,Ben R Newell,Brian A Nosek,Russell A Poldrack,Don van Ravenzwaaij,Jörg Rieskamp,Matthew J Salganik,Alexandra Sarafoglou,Tom Schonberg,Martin Schweinsberg,David Shanks,Raphael Silberzahn,Daniel J Simons,Barbara A Spellman,Samuel St-Jean,Jeffrey J Starns,Eric Luis Uhlmann,Jelte Wicherts,Eric-Jan Wagenmakers,Noah N N van Dongen,Ann Dreber

DOI: https://doi.org/10.7554/eLife.72185

IF: 7.7

2021-11-09

eLife

Abstract:Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

biology
Closure properties of classes of multiple testing procedures

Georg Hahn

DOI: https://doi.org/10.1007/s10182-017-0297-0

2017-04-30

Abstract:Statistical discoveries are often obtained through multiple hypothesis testing. A variety of procedures exists to evaluate multiple hypotheses, for instance the ones of Benjamini-Hochberg, Bonferroni, Holm or Sidak. We are particularly interested in multiple testing procedures with two desired properties: (solely) monotonic and well-behaved procedures. This article investigates to which extent the classes of (monotonic or well-behaved) multiple testing procedures, in particular the subclasses of so-called step-up and step-down procedures, are closed under basic set operations, specifically the union, intersection, difference and the complement of sets of rejected or non-rejected hypotheses. The present article proves two main results: First, taking the union or intersection of arbitrary (monotonic or well-behaved) multiple testing procedures results in new procedures which are monotonic but not well-behaved, whereas the complement or difference generally preserves neither property. Second, the two classes of (solely monotonic or well-behaved) step-up and step-down procedures are closed under taking the union or intersection, but not the complement or difference.

Statistics Theory
Rejoinder: New Objectives for Policy Learning

Nathan Kallus

DOI: https://doi.org/10.48550/arXiv.2012.03130

2020-12-06

Abstract:I provide a rejoinder for discussion of "More Efficient Policy Learning via Optimal Retargeting" to appear in the Journal of the American Statistical Association with discussion by Oliver Dukes and Stijn Vansteelandt; Sijia Li, Xiudi Li, and Alex Luedtkeand; and Muxuan Liang and Yingqi Zhao.

Machine Learning,Optimization and Control
Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions

Jelle Goeman,Jesse Hemerik,Aldo Solari

DOI: https://doi.org/10.1214/20-AOS1999

2022-04-29

Abstract:We consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class encompasses methods controlling familywise error rate, generalized familywise error rate, false discovery exceedance, joint error rate, simultaneous control of all false discovery proportions, and others, as well as seemingly unrelated methods such as gene set testing in genomics and cluster inference methods in neuroimaging. We show that all such methods are either equivalent to a closed testing method, or are uniformly improved by one. Moreover, we show that a closed testing method is admissible as a method controlling tail probabilities of false discovery proportions if and only if all its local tests are admissible. This implies that, when designing such methods, it is sufficient to restrict attention to closed testing methods only. We demonstrate the practical usefulness of this design principle by constructing a uniform improvement of a recently proposed method.

Statistics Theory,Methodology

Rejoinder to "Multiple Testing for Exploratory Research"

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Testing Hypotheses When You Have More Than a Few

Rejoinder to Article by Berger, Bernardo, and Sun

MultipleTesting.com: a tool for life science researchers for multiple hypothesis testing correction

Dealing with multiple testing: To adjust or not to adjust

Multiple testing in statistical analysis of systems-based information retrieval experiments

More Powerful Multiple Testing in Randomized Experiments with Non-Compliance

A model of multiple hypothesis testing

A Practitioner's Guide to Multiple Testing Error Rates

Rejoinder: Struggles with Survey Weighting and Regression Modeling

On particle acceleration and trapping by Poynting flux dominated flows

A Family of Bayes Multiple Testing Procedures

What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research

The sequential rejection principle of familywise error control

A New Multiple Testing Method in the Dependent Case

Addressing researcher degrees of freedom through minP adjustment

Consensus-based guidance for conducting and reporting multi-analyst studies

Closure properties of classes of multiple testing procedures

Rejoinder: New Objectives for Policy Learning

Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions