Abstract:Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $\gamma$ and $\alpha$, $P\{FDP>\gamma\}\le \alpha$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

GMM-based Procedure for Multiple Hypotheses Testing

Multiple Testing with Heterogeneous Multinomial Distributions

A Powerful Fdr Control Procedure for Multiple Hypotheses

Optimal False Discovery Rate Control for Large Scale Multiple Testing with Auxiliary Information

Optimal control of false discovery criteria in the two-group model

Large-scale Multiple Testing: Fundamental Limits of False Discovery Rate Control and Compound Oracle

Control of generalized error rates in multiple testing

Testing a Large Number of Composite Null Hypotheses Using Conditionally Symmetric Multidimensional Gaussian Mixtures in Genome-Wide Studies

A grouped, selectively weighted false discovery rate procedure

Generalized FWER Control Procedures for Testing Multiple Hypotheses

Two-stage Hypothesis Tests for Variable Interactions with FDR Control

F-distribution Calibrated Empirical Likelihood Ratio Tests for Multiple Hypothesis Testing

False Discovery Control in Multiple Testing: A Brief Overview of Theories and Methodologies

Accumulation tests for FDR control in ordered hypothesis testing

A New Procedure for Controlling False Discovery Rate in Large-Scale t-tests

Mixed Directional False Discovery Rate Control in Multiple Pairwise Comparisons Using Weighted P‐values

Multiple-Testing Procedure Based on Generalized Inference of True Null Hypotheses Proportion

Weighted p -value procedures for controlling FDR of grouped hypotheses

Gaussian Graphical Model Estimation with False Discovery Rate Control

Control of False Discoveries in Grouped Hypothesis Testing for Eqtl Data

Multiple Testing in Nonparametric Hidden Markov Models: An Empirical Bayes Approach