Abstract:Latent Dirichlet allocation (LDA) is a topic model widely used for discovering hidden semantics in massive text corpora. Collapsed Gibbs sampling (CGS), as a widely-used algorithm for learning the parameters of LDA, has the risk of privacy leakage. Specifically, word count statistics and updates of latent topics in CGS, which are essential for parameter estimation, could be employed by adversaries to conduct effective membership inference attacks (MIAs). Till now, there are two kinds of methods exploited in CGS to defend against MIAs: adding noise to word count statistics and utilizing inherent privacy. These two kinds of methods have their respective limitations. Noise sampled from the Laplacian distribution sometimes produces negative word count statistics, which render terrible parameter estimation in CGS. Utilizing inherent privacy could only provide weak guaranteed privacy when defending against MIAs. It is promising to propose an effective framework to obtain accurate parameter estimations with guaranteed differential privacy. The key issue of obtaining accurate parameter estimations when introducing differential privacy in CGS is making good use of the privacy budget such that a precise noise scale is derived. It is the first time that Rényi differential privacy (RDP) has been introduced into CGS and we propose RDP-LDA, an effective framework for analyzing the privacy loss of any differentially private CGS. RDP-LDA could be used to derive a tighter upper bound of privacy loss than the overestimated results of existing differentially private CGS obtained by ε -DP. In RDP-LDA, we propose a novel truncated-Gaussian mechanism that keeps word count statistics non-negative. And we propose distribution perturbation which could provide more rigorous guaranteed privacy than utilizing inherent privacy. Experiments validate that our proposed methods produce more accurate parameter estimation under the JS-divergence metric and obtain lower precision and recall when defending against MIAs.

Data Augmentation MCMC for Bayesian Inference from Privatized Data

Statistical Inference for Privatized Data with Unknown Sample Size

Differentially Private Variational Inference for Non-conjugate Models

Differentially Private Distributed Bayesian Linear Regression with MCMC

Conditional Density Estimations from Privacy-Protected Data

Optimal Local Bayesian Differential Privacy over Markov Chains

Exact and Efficient Bayesian Inference for Privacy Risk Quantification (Extended Version)

Tailoring Differentially Private Bayesian Inference to Distance Between Distributions

MACE: A Flexible Framework for Membership Privacy Estimation in Generative Models

Secure and Differentially Private Bayesian Learning on Distributed Data

Simulation-based, Finite-sample Inference for Privatized Data

Bayesian Inference Under Differential Privacy With Bounded Data

Equation Chapter 1 Section 1 Differentially Private High-Dimensional Binary Data Publication via Adaptive Bayesian Network

PrivGMM: Probability Density Estimation with Local Differential Privacy

Differentially Private Bayesian Optimization

On the Use of Penalty MCMC for Differential Privacy

Privacy Budgeting for Growing Machine Learning Datasets

Improving Parameter Estimation and Defensive Ability of Latent Dirichlet Allocation Model Training under Rényi Differential Privacy

Bayesian Frequency Estimation Under Local Differential Privacy With an Adaptive Randomized Response Mechanism

Tractable MCMC for Private Learning with Pure and Gaussian Differential Privacy

Differentially Private Model Selection with Penalized and Constrained Likelihood