A more practical approach for the Benjamini-Hochberg FDR controlling procedure for huge-scale testing problems

Vered Madar,Sandra Batista

DOI: https://doi.org/10.48550/arXiv.1501.05225

2015-01-22

Abstract:We address a common problem in large-scale data analysis, and especially the field of genetics, the huge-scale testing problem, where millions to billions of hypotheses are tested together creating a computational challenge to perform multiple hypotheses testing procedures. As a solution we propose an alternative algorithm to the well used Linear Step Up procedure of Benjamini and Hochberg (1995). Our algorithm requires linear time and does not require any p-value ordering. It permits separating huge-scale testing problems arbitrarily into computationally feasible sets or chunks. Results from the chunks are combined by our algorithm to produce the same results as the controlling procedure on the entire set of tests, thus controlling the global false discovery rate even when p-values are arbitrarily divided. The practical memory usage may also be determined arbitrarily by the size of available memory.

Methodology