Abstract:Recent success of deep neural networks (DNNs) hinges on the availability of large-scale dataset; however, training on such dataset often poses privacy risks for sensitive training information. In this paper, we aim to explore the power of generative models and gradient sparsity, and propose a scalable privacy-preserving generative model DataLens, which is able to generate synthetic data in a differentially private (DP) way given sensitive input data. Thus, it is possible to train models for different down-stream tasks with the generated data while protecting the private information. In particular, we leverage the generative adversarial networks (GAN) and PATE framework to train multiple discriminators as "teacher" models, allowing them to vote with their gradient vectors to guarantee privacy. Comparing with the standard PATE privacy preserving framework which allows teachers to vote on one-dimensional predictions, voting on the high dimensional gradient vectors is challenging in terms of privacy preservation. As dimension reduction techniques are required, we need to navigate a delicate tradeoff space between (1) the improvement of privacy preservation and (2) the slowdown of SGD convergence. To tackle this, we propose a novel dimension compression and aggregation approach TopAgg, which combines top-k dimension compression with a corresponding noise injection mechanism. We theoretically prove that the DataLens framework guarantees differential privacy for its generated data, and provide a novel analysis on its convergence to illustrate such a tradeoff on privacy and convergence rate, which requires non-trivial analysis as it requires a joint analysis on gradient compression, coordinate-wise gradient clipping, and DP mechanism. To demonstrate the practical usage of DataLens, we conduct extensive experiments on diverse datasets including MNIST, Fashion-MNIST, and high dimensional CelebA and Place365 datasets. We show that DataLens significantly outperforms other baseline differentially private data generative models. Our code is publicly available at https://github.com/AI-secure/DataLens.

Multi-modal AsynDGAN: Learn From Distributed Medical Image Data without Sharing Private Information

Synthetic learning: Learn from distributed asynchronized discriminator gan without sharing medical image data

Privacy-Preserving Collaborative Deep Learning with Unreliable Participants.

PKDGAN: Private Knowledge Distillation with Generative Adversarial Networks

Ensemble-Based Distributed Learning for Generative Adversarial Networks.

A New Distributed Method for Training Generative Adversarial Networks

Modality Bank: Learn multi-modality images across data centers without sharing medical data

Mining multi-center heterogeneous medical data with distributed synthetic learning

AutoEncoder-Driven Multimodal Collaborative Learning for Medical Image Synthesis

Enabling Competitive Performance of Medical Imaging with Diffusion Model-generated Images without Privacy Leakage

Federated Edge Learning for Medical Image Augmentation

Federated Synthetic Data Generation with Differential Privacy

Constrained generative adversarial network ensembles for sharable synthetic medical images

Privacy enhancing and generalizable deep learning with synthetic data for mediastinal neoplasm diagnosis

MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets

DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation

Learning Geographically Distributed Data for Multiple Tasks Using Generative Adversarial Networks.

Auto-GAN: Self-Supervised Collaborative Learning for Medical Image Synthesis.

Distributed deep learning networks among institutions for medical imaging

GH-DDM: the generalized hybrid denoising diffusion model for medical image generation

Distributed Generative Adversarial Net