Abstract:Synthetic data generation is being increasingly used as a privacy preserving approach for sharing health data. In addition to protecting privacy, it is important to ensure that generated data has high utility. A common way to assess utility is the ability of synthetic data to replicate results from the real data. Replicability has been defined using two criteria: (a) replicate the results of the analyses on real data, and (b) ensure valid population inferences from the synthetic data. A simulation study using three heterogeneous real-world datasets evaluated the replicability of logistic regression workloads. Eight replicability metrics were evaluated: decision agreement, estimate agreement, standardized difference, confidence interval overlap, bias, confidence interval coverage, statistical power, and precision (empirical SE). The analysis of synthetic data used a multiple imputation approach whereby up to 20 datasets were generated and the fitted logistic regression models were combined using combining rules for fully synthetic datasets. The effects of synthetic data amplification were evaluated, and two types of generative models were used: sequential synthesis using boosted decision trees and a generative adversarial network (GAN). Privacy risk was evaluated using a membership disclosure metric. For sequential synthesis, adjusted model parameters after combining at least ten synthetic datasets gave high decision and estimate agreement, low standardized difference, as well as high confidence interval overlap, low bias, the confidence interval had nominal coverage, and power close to the nominal level. Amplification had only a marginal benefit. Confidence interval coverage from a single synthetic dataset without applying combining rules were erroneous, and statistical power, as expected, was artificially inflated when amplification was used. Sequential synthesis performed considerably better than the GAN across multiple datasets. Membership disclosure risk was low for all datasets and models. For replicable results, the statistical analysis of fully synthetic data should be based on at least ten generated datasets of the same size as the original whose analyses results are combined. Analysis results from synthetic data without applying combining rules can be misleading. Replicability results are dependent on the type of generative model used, with our study suggesting that sequential synthesis has good replicability characteristics for common health research workloads.

Comparing basic statistical concepts with diagnostic probabilities based on directly observed proportions to help understand the replication crisis

A new standard for the analysis and design of replication studies

Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Statistics in Service of Metascience: Measuring Replication Distance with Reproducibility Rate

Assessing replication success via skeptical mixture priors

The assessment of replicability using the sum of p-values

Analysis of Bayesian posterior significance and effect size indices for the two-sample t-test to support reproducible medical research

Statistical reform and the replication crisis

The Replication of Equivalence Studies

Estimating Reproducibility in Genome-Wide Association Studies

Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Can the Replication Rate Tell Us About Publication Bias?

Statistical Assessment of Replicability via Bayesian Model Criticism

Replicability analysis for genome-wide association studies

Estimating the reproducibility of psychological science

An evaluation of the replicability of analyses using synthetic health data

Large Scale Replication Projects in Contemporary Psychological Research

Eliminating Accidental Deviations to Minimize Generalization Error and Maximize Replicability: Applications in Connectomics and Genomics.

Replication of null results: Absence of evidence or evidence of absence?

Randomized p-values for multiple testing and their application in replicability analysis