Abstract:Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $\gamma$ and $\alpha$, $P\{FDP>\gamma\}\le \alpha$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Multiple testing of response rates with a control: a Bayesian stepwise approach

A Family of Bayes Multiple Testing Procedures

Bayesian Multivariate Probability of Success Using Historical Data with Strict Control of Family-wise Error Rate

A Bayesian Platform Trial Design to Simultaneously Evaluate Multiple Drugs in Multiple Indications with Mixed Endpoints.

A Bayesian Design for Phase II Clinical Trials with Delayed Responses Based on Multiple Imputation.

A New Multiple Testing Method in the Dependent Case

Non-marginal Decisions: A Novel Bayesian Multiple Testing Procedure

Generalized FWER Control Procedures for Testing Multiple Hypotheses

A Bayesian approach for evaluating equivalence over multiple groups, and comparison with frequentist tost

A model of multiple hypothesis testing

Comparisons of methods for multiple hypothesis testing in neuropsychological research.

Control of generalized error rates in multiple testing

Empirical partially Bayes multiple testing and compound $χ^2$ decisions

Testing the equality of response rate functions for paired binary data with multiple groups

A multiple testing framework for diagnostic accuracy studies with co-primary endpoints

Multiple Testing in Nonparametric Hidden Markov Models: An Empirical Bayes Approach

Bayesian analysis for pretest-posttest binary outcomes with adaptive significance levels

Optimal exact tests for multiple binary endpoints

More Powerful Multiple Testing in Randomized Experiments with Non-Compliance

A class of sequential multi-hypothesis tests

On Controlling the False Discovery Rate in Multiple Testing of the Means of Correlated Normals Against Two-Sided Alternatives