Abstract:Adaptive data analysis has posed a challenge to science due to its ability to generate false hypotheses on moderately large data sets. In general, with non-adaptive data analyses (where queries to the data are generated without being influenced by answers to previous queries) a data set containing $n$ samples may support exponentially many queries in $n$. This number reduces to linearly many under naive adaptive data analysis, and even sophisticated remedies such as the Reusable Holdout (Dwork et. al 2015) only allow quadratically many queries in $n$. In this work, we propose a new framework for adaptive science which exponentially improves on this number of queries under a restricted yet scientifically relevant setting, where the goal of the scientist is to find a single (or a few) true hypotheses about the universe based on the samples. Such a setting may describe the search for predictive factors of some disease based on medical data, where the analyst may wish to try a number of predictive models until a satisfactory one is found. Our solution, the Generic Holdout, involves two simple ingredients: (1) a partitioning of the data into a exploration set and a holdout set and (2) a limited exposure strategy for the holdout set. An analyst is free to use the exploration set arbitrarily, but when testing hypotheses against the holdout set, the analyst only learns the answer to the question: "Is the given hypothesis true (empirically) on the holdout set?" -- and no more information, such as "how well" the hypothesis fits the holdout set. The resulting scheme is immediate to analyze, but despite its simplicity we do not believe our method is obvious, as evidenced by the many violations in practice. Our proposal can be seen as an alternative to pre-registration, and allows researchers to get the benefits of adaptive data analysis without the problems of adaptivity.

Raiders of the Lost HARK: a Reproducible Inference Framework for Big Data Science

Statistical inference on representational geometries

Detecting signal from science:The structure of research communities and prior knowledge improves prediction of genetic regulatory experiments

The Generic Holdout: Preventing False-Discoveries in Adaptive Data Science

Navigating the Landscape of Reproducible Research: A Predictive Modeling Approach

A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results

The Causal Roadmap and Simulations to Improve the Rigor and Reproducibility of Real-data Applications

Smoke and Mirrors in Causal Downstream Tasks

posteriordb: Testing, Benchmarking and Developing Bayesian Inference Algorithms

Inferactive data analysis

Predicting and reasoning about replicability using structured groups

Irreproducibility; Nothing is More Predictable

A Case Study on a Sustainable Framework for Ethically Aware Predictive Modeling

Causal Inference in the Age of Big Data: Blind Faith in Data and Technology

Modeling Sustainable Resource Management using Active Inference

Hidden data recovery using reservoir computing: Adaptive network model and experimental brain signals

Distilling Multi-Scale Neural Mechanisms from Diverse Unlabeled Experimental Data Using Deep Domain-Adaptive Inference Framework

Credible causal inference beyond toy models

Practical Bayesian Inference in Neuroscience: Or How I Learned To Stop Worrying and Embrace the Distribution

Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

Statistical Inference for Fairness Auditing