Abstract:Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $\gamma$ and $\alpha$, $P\{FDP>\gamma\}\le \alpha$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Doublethink: simultaneous Bayesian-frequentist model-averaged hypothesis testing

A frequentist two-sample test based on Bayesian model selection

Model-averaged Bayesian t tests

Identifying direct risk factors in UK Biobank with simultaneous Bayesian-frequentist model-averaged hypothesis testing using Doublethink

A new approach to Bayesian hypothesis testing

Simultaneous inference: When should hypothesis testing problems be combined?

Consistent estimation of the proportion of false nulls and FDR for adaptive multiple testing Normal means under weak dependence

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Multiple testing with the structure adaptive Benjamini-Hochberg algorithm

Multiple Testing in Nonparametric Hidden Markov Models: An Empirical Bayes Approach

Compound e-values and empirical Bayes

Smaller $p$-values in genomics studies using distilled historical information

The flaw of averages: Bayes factors as posterior means of the likelihood ratio

Comparing researchers' degree of dichotomous thinking using frequentist versus Bayesian null hypothesis testing

Changing the paradigm of fixed significance levels: Testing Hypothesis by Minimizing Sum of Errors Type I and Type II

Statistical significance testing for mixed priors: a combined Bayesian and frequentist analysis

Empirical partially Bayes multiple testing and compound $χ^2$ decisions

Sharp Results for Hypothesis Testing with Risk-Sensitive Agents

Control of generalized error rates in multiple testing

A Good check on the Bayes factor

Empirical Bayes large-scale multiple testing for high-dimensional binary outcome data