Abstract:Latent Dirichlet allocation (LDA) is a topic model widely used for discovering hidden semantics in massive text corpora. Collapsed Gibbs sampling (CGS), as a widely-used algorithm for learning the parameters of LDA, has the risk of privacy leakage. Specifically, word count statistics and updates of latent topics in CGS, which are essential for parameter estimation, could be employed by adversaries to conduct effective membership inference attacks (MIAs). Till now, there are two kinds of methods exploited in CGS to defend against MIAs: adding noise to word count statistics and utilizing inherent privacy. These two kinds of methods have their respective limitations. Noise sampled from the Laplacian distribution sometimes produces negative word count statistics, which render terrible parameter estimation in CGS. Utilizing inherent privacy could only provide weak guaranteed privacy when defending against MIAs. It is promising to propose an effective framework to obtain accurate parameter estimations with guaranteed differential privacy. The key issue of obtaining accurate parameter estimations when introducing differential privacy in CGS is making good use of the privacy budget such that a precise noise scale is derived. It is the first time that Rényi differential privacy (RDP) has been introduced into CGS and we propose RDP-LDA, an effective framework for analyzing the privacy loss of any differentially private CGS. RDP-LDA could be used to derive a tighter upper bound of privacy loss than the overestimated results of existing differentially private CGS obtained by ε -DP. In RDP-LDA, we propose a novel truncated-Gaussian mechanism that keeps word count statistics non-negative. And we propose distribution perturbation which could provide more rigorous guaranteed privacy than utilizing inherent privacy. Experiments validate that our proposed methods produce more accurate parameter estimation under the JS-divergence metric and obtain lower precision and recall when defending against MIAs.

PrivGMM: Probability Density Estimation with Local Differential Privacy

DPHMM: Customizable Data Release with Differential Privacy Via Hidden Markov Model.

GANs Based Density Distribution Privacy-Preservation on Mobility Data

Conditional Density Estimations from Privacy-Protected Data

Generalized Gaussian Mechanism for Differential Privacy

Improving Parameter Estimation and Defensive Ability of Latent Dirichlet Allocation Model Training under Rényi Differential Privacy

Locally Private Gaussian Estimation

Securely Sampling Discrete Gaussian Noise for Multi-Party Differential Privacy

Differentially Private Variational Inference for Non-conjugate Models

Instance-optimal Mean Estimation Under Differential Privacy

Adaptive pointwise density estimation under local differential privacy

Differentially private sub-Gaussian location estimators

Differentially Private Robust ADMM for Distributed Machine Learning.

Improved Matrix Gaussian Mechanism for Differential Privacy

Numerical Estimation of Spatial Distributions under Differential Privacy

Tractable MCMC for Private Learning with Pure and Gaussian Differential Privacy

Differential Privacy for Class-based Data: A Practical Gaussian Mechanism

Data Augmentation MCMC for Bayesian Inference from Privatized Data

The Discrete Gaussian for Differential Privacy

A Differential Privacy Mechanism Design Under Matrix-Valued Query

Efficient Estimation of a Gaussian Mean with Local Differential Privacy