Abstract:Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $\gamma$ and $\alpha$, $P\{FDP>\gamma\}\le \alpha$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses

Two New Estimators for the Proportion of True Null Hypotheses in Multiple Test

Consistent estimation of the proportion of false nulls and FDR for adaptive multiple testing Normal means under weak dependence

A Change-Point Approach to Estimating the Proportion of False Null Hypotheses in Multiple Testing

Optimal False Discovery Rate Control for Large Scale Multiple Testing with Auxiliary Information

Estimating the proportion of true null hypotheses and adaptive false discovery rate control in discrete paradigm

Estimating the number and effect sizes of non-null hypotheses

Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing

Estimation of Proportion of Null Hypotheses Under Dependence

Confidence bounds for the true discovery proportion based on the exact distribution of the number of rejections

GMM-based Procedure for Multiple Hypotheses Testing

Flexible control of the median of the false discovery proportion

Estimating the null distribution for conditional inference and genome-scale screening

Control of generalized error rates in multiple testing

On stepdown control of the false discovery proportion

Interval estimation, point estimation, and null hypothesis significance testing calibrated by an estimated posterior probability of the null hypothesis

Estimating false discovery proportion in multiple comparison under dependency.

Multiple-Testing Procedure Based on Generalized Inference of True Null Hypotheses Proportion

Confidence distributions and hypothesis testing

Error-rate and decision-theoretic methods of multiple testing: Which genes have high objective probabilities of differential expression?

General ways to improve false coverage rate-adjusted selective confidence intervals