Abstract:Synthetic data generation is being increasingly used as a privacy preserving approach for sharing health data. In addition to protecting privacy, it is important to ensure that generated data has high utility. A common way to assess utility is the ability of synthetic data to replicate results from the real data. Replicability has been defined using two criteria: (a) replicate the results of the analyses on real data, and (b) ensure valid population inferences from the synthetic data. A simulation study using three heterogeneous real-world datasets evaluated the replicability of logistic regression workloads. Eight replicability metrics were evaluated: decision agreement, estimate agreement, standardized difference, confidence interval overlap, bias, confidence interval coverage, statistical power, and precision (empirical SE). The analysis of synthetic data used a multiple imputation approach whereby up to 20 datasets were generated and the fitted logistic regression models were combined using combining rules for fully synthetic datasets. The effects of synthetic data amplification were evaluated, and two types of generative models were used: sequential synthesis using boosted decision trees and a generative adversarial network (GAN). Privacy risk was evaluated using a membership disclosure metric. For sequential synthesis, adjusted model parameters after combining at least ten synthetic datasets gave high decision and estimate agreement, low standardized difference, as well as high confidence interval overlap, low bias, the confidence interval had nominal coverage, and power close to the nominal level. Amplification had only a marginal benefit. Confidence interval coverage from a single synthetic dataset without applying combining rules were erroneous, and statistical power, as expected, was artificially inflated when amplification was used. Sequential synthesis performed considerably better than the GAN across multiple datasets. Membership disclosure risk was low for all datasets and models. For replicable results, the statistical analysis of fully synthetic data should be based on at least ten generated datasets of the same size as the original whose analyses results are combined. Analysis results from synthetic data without applying combining rules can be misleading. Replicability results are dependent on the type of generative model used, with our study suggesting that sequential synthesis has good replicability characteristics for common health research workloads.

Synthetic Census Microdata Generation: A Comparative Study of Synthesis Methods Examining the Trade-Off Between Disclosure Risk and Utility

Generative Adversarial Networks for Synthetic Data Generation: A Comparative Study

Advancing microdata privacy protection: A review of synthetic data methods

Utility Assessment of Synthetic Data Generation Methods

GenSyn: A Multi-stage Framework for Generating Synthetic Microdata using Macro Data Sources

Statistical properties and privacy guarantees of an original distance-based fully synthetic data generation method

Synthetic Census Data Generation via Multidimensional Multiset Sum

Synthesizing geocodes to facilitate access to detailed geographical information in large scale administrative data

Bayesian Data Synthesis and Disclosure Risk Quantification: An Application to the Consumer Expenditure Surveys

Evaluating utility in synthetic banking microdata applications

Multiply-Imputed Synthetic Data: Advice to the Imputer

Fake It Till You Make It: Guidelines for Effective Synthetic Data Generation

30 Years of Synthetic Data

Boosting Data Analytics With Synthetic Volume Expansion

On Utility and Privacy in Synthetic Genomic Data

Artificial Data, Real Insights: Evaluating Opportunities and Risks of Expanding the Data Ecosystem with Synthetic Data

Comparative Study of Differentially Private Synthetic Data Algorithms from the NIST PSCR Differential Privacy Synthetic Data Challenge

A density ratio framework for evaluating the utility of synthetic data

Using saturated count models for user-friendly synthesis of categorical data

Synthetic Generation of Patient Service Utilization Data: A Scalability Study

An evaluation of the replicability of analyses using synthetic health data