Abstract:Synthetic data generation is being increasingly used as a privacy preserving approach for sharing health data. In addition to protecting privacy, it is important to ensure that generated data has high utility. A common way to assess utility is the ability of synthetic data to replicate results from the real data. Replicability has been defined using two criteria: (a) replicate the results of the analyses on real data, and (b) ensure valid population inferences from the synthetic data. A simulation study using three heterogeneous real-world datasets evaluated the replicability of logistic regression workloads. Eight replicability metrics were evaluated: decision agreement, estimate agreement, standardized difference, confidence interval overlap, bias, confidence interval coverage, statistical power, and precision (empirical SE). The analysis of synthetic data used a multiple imputation approach whereby up to 20 datasets were generated and the fitted logistic regression models were combined using combining rules for fully synthetic datasets. The effects of synthetic data amplification were evaluated, and two types of generative models were used: sequential synthesis using boosted decision trees and a generative adversarial network (GAN). Privacy risk was evaluated using a membership disclosure metric. For sequential synthesis, adjusted model parameters after combining at least ten synthetic datasets gave high decision and estimate agreement, low standardized difference, as well as high confidence interval overlap, low bias, the confidence interval had nominal coverage, and power close to the nominal level. Amplification had only a marginal benefit. Confidence interval coverage from a single synthetic dataset without applying combining rules were erroneous, and statistical power, as expected, was artificially inflated when amplification was used. Sequential synthesis performed considerably better than the GAN across multiple datasets. Membership disclosure risk was low for all datasets and models. For replicable results, the statistical analysis of fully synthetic data should be based on at least ten generated datasets of the same size as the original whose analyses results are combined. Analysis results from synthetic data without applying combining rules can be misleading. Replicability results are dependent on the type of generative model used, with our study suggesting that sequential synthesis has good replicability characteristics for common health research workloads.

Predicting replicability -- analysis of survey and prediction market data from large-scale forecasting projects

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study

Irreproducibility in Searches of Scientific Literature: A Comparative Analysis.

Examining the replicability of online experiments selected by a decision market

Predicting and reasoning about replicability using structured groups

Estimating the reproducibility of psychological science

Prediction scoring of data-driven discoveries for reproducible research

Replicability of simulation studies for the investigation of statistical methods: the RepliSims project

A prototype hybrid prediction market for estimating replicability of published work

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Reproducibility of prediction models in health services research

Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models

Reproducibility Signals in Science: A preliminary analysis

Statistical Methods for Replicability Assessment

A glass half full interpretation of the replicability of psychological science

An evaluation of the replicability of analyses using synthetic health data

Large Scale Replication Projects in Contemporary Psychological Research

A new standard for the analysis and design of replication studies

Data from a Pre-Publication Independent Replication Initiative Examining Ten Moral Judgement Effects