Abstract:Latent Dirichlet allocation (LDA) is a topic model widely used for discovering hidden semantics in massive text corpora. Collapsed Gibbs sampling (CGS), as a widely-used algorithm for learning the parameters of LDA, has the risk of privacy leakage. Specifically, word count statistics and updates of latent topics in CGS, which are essential for parameter estimation, could be employed by adversaries to conduct effective membership inference attacks (MIAs). Till now, there are two kinds of methods exploited in CGS to defend against MIAs: adding noise to word count statistics and utilizing inherent privacy. These two kinds of methods have their respective limitations. Noise sampled from the Laplacian distribution sometimes produces negative word count statistics, which render terrible parameter estimation in CGS. Utilizing inherent privacy could only provide weak guaranteed privacy when defending against MIAs. It is promising to propose an effective framework to obtain accurate parameter estimations with guaranteed differential privacy. The key issue of obtaining accurate parameter estimations when introducing differential privacy in CGS is making good use of the privacy budget such that a precise noise scale is derived. It is the first time that Rényi differential privacy (RDP) has been introduced into CGS and we propose RDP-LDA, an effective framework for analyzing the privacy loss of any differentially private CGS. RDP-LDA could be used to derive a tighter upper bound of privacy loss than the overestimated results of existing differentially private CGS obtained by ε -DP. In RDP-LDA, we propose a novel truncated-Gaussian mechanism that keeps word count statistics non-negative. And we propose distribution perturbation which could provide more rigorous guaranteed privacy than utilizing inherent privacy. Experiments validate that our proposed methods produce more accurate parameter estimation under the JS-divergence metric and obtain lower precision and recall when defending against MIAs.

Differentially Private Density Estimation Via Gaussian Mixtures Model

PrivGMM: Probability Density Estimation with Local Differential Privacy

DPHMM: Customizable Data Release with Differential Privacy Via Hidden Markov Model.

Private Estimation with Public Data

p-Power Exponential Mechanisms for Differentially Private Machine Learning

DP-EM: Differentially Private Expectation Maximization

Matrix Gaussian Mechanisms for Differentially-Private Learning

Learning Probability Density Functions from Marginal Distributions with Applications to Gaussian Mixtures

Improving Parameter Estimation and Defensive Ability of Latent Dirichlet Allocation Model Training under Rényi Differential Privacy

Generalized Gaussian Mechanism for Differential Privacy

Securely Sampling Discrete Gaussian Noise for Multi-Party Differential Privacy

Tractable MCMC for Private Learning with Pure and Gaussian Differential Privacy

Conditional Density Estimations from Privacy-Protected Data

Selecting Optimal Subset to release under Differentially Private M-estimators from Hybrid Datasets

Auditing Differential Privacy Guarantees Using Density Estimation

Differentially Private Synthetic Data with Private Density Estimation

Adaptive pointwise density estimation under local differential privacy

Differentially Private Robust ADMM for Distributed Machine Learning.

Differentially Private Regression and Classification with Sparse Gaussian Processes

Sample-Efficient Private Learning of Mixtures of Gaussians

Differential Privacy for Class-based Data: A Practical Gaussian Mechanism