Abstract:In a multiple testing problem where one is willing to tolerate a few false rejections, procedure controlling the familywise error rate (FWER) can potentially be improved in terms of its ability to detect false null hypotheses by generalizing it to control the $k$-FWER, the probability of falsely rejecting at least $k$ null hypotheses, for some fixed $k>1$. Simes' test for testing the intersection null hypothesis is generalized to control the $k$-FWER weakly, that is, under the intersection null hypothesis, and Hochberg's stepup procedure for simultaneous testing of the individual null hypotheses is generalized to control the $k$-FWER strongly, that is, under any configuration of the true and false null hypotheses. The proposed generalizations are developed utilizing joint null distributions of the $k$-dimensional subsets of the $p$-values, assumed to be identical. The generalized Simes' test is proved to control the $k$-FWER weakly under the multivariate totally positive of order two (MTP$_2$) condition [J. Multivariate Analysis 10 (1980) 467--498] of the joint null distribution of the $p$-values by generalizing the original Simes' inequality. It is more powerful to detect $k$ or more false null hypotheses than the original Simes' test when the $p$-values are independent. A stepdown procedure strongly controlling the $k$-FWER, a version of generalized Holm's procedure that is different from and more powerful than [Ann. Statist. 33 (2005) 1138--1154] with independent $p$-values, is derived before proposing the generalized Hochberg's procedure. The strong control of the $k$-FWER for the generalized Hochberg's procedure is established in situations where the generalized Simes' test is known to control its $k$-FWER weakly.

The one-sided posterior predictive p-value for Fieller’s problem

Fiducialize statistical significance: transforming p-values into conservative posterior probabilities and Bayes factors

Generalizing Simes' test and Hochberg's stepup procedure

Consistent estimation of the proportion of false nulls and FDR for adaptive multiple testing Normal means under weak dependence

P-value: A Bless or A Curse for Evidence-Based Studies?

Asymptotic false discovery control of the Benjamini-Hochberg procedure for pairwise comparisons

Multiple testing of composite null hypotheses for discrete data using randomized $p$-values

A Change-Point Approach to Estimating the Proportion of False Null Hypotheses in Multiple Testing

Post-hoc Hypothesis Testing

Interval estimation, point estimation, and null hypothesis significance testing calibrated by an estimated posterior probability of the null hypothesis

Integrative conformal p -values for out-of-distribution testing with labelled outliers

Confidence distributions and hypothesis testing

The flaw of averages: Bayes factors as posterior means of the likelihood ratio

Asymptotics for Outlier Hypothesis Testing

On a generalized false discovery rate

Adaptative significance levels in normal mean hypothesis testing

Empirical Bayes large-scale multiple testing for high-dimensional binary outcome data

Selective inference is easier with p-values

Flexible control of the median of the false discovery proportion

A grouped, selectively weighted false discovery rate procedure

On two-sided p-values for non-symmetric distributions