Abstract:<h3>Background</h3><p>In high-throughput studies, hundreds to millions of hypotheses are typically tested. Statistical methods that control the false discovery rate (FDR) have emerged as popular and powerful tools for error rate control. While classic FDR methods use only <em>p</em> values as input, more modern FDR methods have been shown to increase power by incorporating complementary information as informative covariates to prioritize, weight, and group hypotheses. However, there is currently no consensus on how the modern methods compare to one another. We investigate the accuracy, applicability, and ease of use of two classic and six modern FDR-controlling methods by performing a systematic benchmark comparison using simulation studies as well as six case studies in computational biology.</p><h3>Results</h3><p>Methods that incorporate informative covariates are modestly more powerful than classic approaches, and do not underperform classic approaches, even when the covariate is completely uninformative. The majority of methods are successful at controlling the FDR, with the exception of two modern methods under certain settings. Furthermore, we find that the improvement of the modern FDR methods over the classic methods increases with the informativeness of the covariate, total number of hypothesis tests, and proportion of truly non-null hypotheses.</p><h3>Conclusions</h3><p>Modern FDR methods that use an informative covariate provide advantages over classic FDR-controlling procedures, with the relative gain dependent on the application and informativeness of available covariates. We present our findings as a practical guide and provide recommendations to aid researchers in their choice of methods to correct for false discoveries.</p>

Estimating false discovery rates under dependence, with applications to DNA microarrays

A statistical method for the conservative adjustment of false discovery rate (q-value)

Computing Power and Sample Size for the False Discovery Rate in Multiple Applications

False discovery rate control: Moving beyond the Benjamini–Hochberg method

Error-rate and decision-theoretic methods of multiple testing: Which genes have high objective probabilities of differential expression?

Selective testing and its effect on false discovery rate controlling procedures under discrete framework

Consistent estimation of the proportion of false nulls and FDR for adaptive multiple testing Normal means under weak dependence

Asymptotic uncertainty of false discovery proportion

Online control of the false discovery rate in biomedical research

Exploratory data analysis for large-scale multiple testing problems and its application in gene expression studies

Estimating the proportion of true null hypotheses and adaptive false discovery rate control in discrete paradigm

High-Dimensional False Discovery Rate Control for Dependent Variables

Optimal False Discovery Rate Control for Large Scale Multiple Testing with Auxiliary Information

Local False Discovery Rate Estimation with Competition-Based Procedures for Variable Selection

A grouped, selectively weighted false discovery rate procedure

A practical guide to methods controlling false discoveries in computational biology

On stepdown control of the false discovery proportion

Asymptotic Uncertainty of False Discovery Proportion for Dependent $t$-Tests

Estimating and Controlling the False Discovery Rate of the PC Algorithm Using Edge-specific P-Values

False Discovery Control in Multiple Testing: A Brief Overview of Theories and Methodologies

A Gene Selection Method for GeneChip Array Data with Small Sample Sizes