Abstract:Background: Synthetic data has been proposed as a solution for sharing anonymized versions of sensitive biomedical datasets. Ideally, synthetic data should preserve the structure and statistical properties of the original data, while protecting the privacy of the individual subjects. Differential privacy (DP) is currently considered the gold standard approach for balancing this trade-off. Objectives: To investigate the reliability of group differences identified by independent sample tests on DP-synthetic data. The evaluation is conducted in terms of the tests' Type I and Type II errors. The former quantifies the tests' validity i.e. whether the probability of false discoveries is indeed below the significance level, and the latter indicates the tests' power in making real discoveries. Methods: We evaluate the Mann-Whitney U test, Student's t-test, chi-squared test and median test on DP-synthetic data. The private synthetic datasets are generated from real-world data, including a prostate cancer dataset (n=500) and a cardiovascular dataset (n=70 000), as well as on bivariate and multivariate simulated data. Five different DP-synthetic data generation methods are evaluated, including two basic DP histogram release methods and MWEM, Private-PGM, and DP GAN algorithms. Conclusion: A large portion of the evaluation results expressed dramatically inflated Type I errors, especially at privacy budget levels of $\epsilon\leq 1$. This result calls for caution when releasing and analyzing DP-synthetic data: low p-values may be obtained in statistical tests simply as a byproduct of the noise added to protect privacy. A DP smoothed histogram-based synthetic data generation method was shown to produce valid Type I error for all privacy levels tested but required a large original dataset size and a modest privacy budget ($\epsilon\geq 5$) in order to have reasonable Type II error.

One Step to Efficient Synthetic Data

Differentially Private Histogram Publication for Dynamic Datasets: an Adaptive Sampling Approach.

PrivSyn: Differentially Private Data Synthesis

The Real Deal Behind the Artificial Appeal: Inferential Utility of Tabular Synthetic Data

Strong statistical parity through fair synthetic data

Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?

Post-processing Private Synthetic Data for Improving Utility on Selected Measures

Synthetic data method to incorporate external information into a current study

Utility Assessment of Synthetic Data Generation Methods

Synthetic Data Approach for Classification and Regression

Differentially Private Low-dimensional Synthetic Data from High-dimensional Datasets

Statistical Theory of Differentially Private Marginal-based Data Synthesis Algorithms

Fake It Till You Make It: Guidelines for Effective Synthetic Data Generation

DP-CDA: An Algorithm for Enhanced Privacy Preservation in Dataset Synthesis Through Randomized Mixing

A density ratio framework for evaluating the utility of synthetic data

Debiasing Synthetic Data Generated by Deep Generative Models

AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Data

Boosting Data Analytics With Synthetic Volume Expansion

Utility Theory of Synthetic Data Generation