Abstract:We propose a general and flexible procedure for testing multiple hypotheses about sequential (or streaming) data that simultaneously controls both the false discovery rate (FDR) and false nondiscovery rate (FNR) under minimal assumptions about the data streams, which may differ in distribution and dimension and be dependent. All that is needed is a test statistic for each data stream that controls its conventional type I and II error probabilities, and no information or assumptions are required about the joint distribution of the statistics or data streams. The procedure can be used with sequential, group sequential, truncated, or other sampling schemes. The procedure is a natural extension of Benjamini and Hochberg's (<a href="#">1995</a> Benjamini, Y. and Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing, Journal of the Royal Statistical Society, Series B: Methodological 57: 289–300. doi:10.1111/j.2517-6161.1995.tb02031.x<a href="/servlet/linkout?suffix=CIT0009&dbid=16&doi=10.1080%2F07474946.2020.1726686&key=10.1111%2Fj.2517-6161.1995.tb02031.x">[Crossref]</a> , <a class="google-scholar" href="http://scholar.google.com/scholar_lookup?hl=en&publication_year=1995&pages=289-300&author=Y.+Benjamini&author=Y.+Hochberg&title=Controlling+the+False+Discovery+Rate%3A+A+Practical+and+Powerful+Approach+to+Multiple+Testing">[Google Scholar]</a>) widely used fixed sample size procedure to the domain of sequential data, with the added benefit of simultaneous FDR and FNR control that sequential sampling affords. We prove the procedure's error control and give some tips for implementation in commonly encountered testing situations.

Sequential tests of multiple hypotheses controlling false discovery and nondiscovery rates

Large-scale adaptive multiple testing for sequential data controlling false discovery and nondiscovery rates

Sequential tests of multiple hypotheses controlling type I and II familywise error rates

False Discovery Control in Multiple Testing: A Brief Overview of Theories and Methodologies

A Rejection Principle for Sequential Tests of Multiple Hypotheses Controlling Familywise Error Rates

Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings

Knockoff procedure for false discovery rate control in high-dimensional data streams

A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence

Online control of the false discovery rate in biomedical research

On stepdown control of the false discovery proportion

On parallel sequential change detection controlling false discovery rate

Large-scale Multiple Testing: Fundamental Limits of False Discovery Rate Control and Compound Oracle

Optimal control of false discovery criteria in the two-group model

A New Procedure for Controlling False Discovery Rate in Large-Scale t-tests

A practical guide to methods controlling false discoveries in computational biology

Online control of the False Discovery Rate in group-sequential platform trials

Non-Bayesian Multiple Change-Point Detection Controlling False Discovery Rate

Power-enhanced multiple decision functions controlling family-wise error and false discovery rates

Multiple testing with the structure adaptive Benjamini-Hochberg algorithm

A more practical approach for the Benjamini-Hochberg FDR controlling procedure for huge-scale testing problems

Null-free False Discovery Rate Control Using Decoy Permutations