Abstract:We consider the problem of learning mixtures of Gaussians under the constraint of approximate differential privacy. We prove that $\widetilde{O}(k^2 d \log^{3/2}(1/\delta) / \alpha^2 \varepsilon)$ samples are sufficient to learn a mixture of $k$ axis-aligned Gaussians in $\mathbb{R}^d$ to within total variation distance $\alpha$ while satisfying $(\varepsilon, \delta)$-differential privacy. This is the first result for privately learning mixtures of unbounded axis-aligned (or even unbounded univariate) Gaussians. If the covariance matrices of each of the Gaussians is the identity matrix, we show that $\widetilde{O}(kd/\alpha^2 + kd \log(1/\delta) / \alpha \varepsilon)$ samples are sufficient. Recently, the "local covering" technique of Bun, Kamath, Steinke, and Wu has been successfully used for privately learning high-dimensional Gaussians with a known covariance matrix and extended to privately learning general high-dimensional Gaussians by Aden-Ali, Ashtiani, and Kamath. Given these positive results, this approach has been proposed as a promising direction for privately learning mixtures of Gaussians. Unfortunately, we show that this is not possible. We design a new technique for privately learning mixture distributions. A class of distributions $\mathcal{F}$ is said to be list-decodable if there is an algorithm that, given "heavily corrupted" samples from $f\in \mathcal{F}$, outputs a list of distributions, $\widehat{\mathcal{F}}$, such that one of the distributions in $\widehat{\mathcal{F}}$ approximates $f$. We show that if $\mathcal{F}$ is privately list-decodable, then we can privately learn mixtures of distributions in $\mathcal{F}$. Finally, we show axis-aligned Gaussian distributions are privately list-decodable, thereby proving mixtures of such distributions are privately learnable.

Approximation of Pufferfish Privacy for Gaussian Priors

Approximation of Pufferfish Privacy for Gaussian Priors

Rényi Pufferfish Privacy: General Additive Noise Mechanisms and Privacy Amplification by Iteration

General Inferential Limits Under Differential and Pufferfish Privacy

Pufferfish Privacy: An Information-Theoretic Study

Count on Your Elders: Laplace vs Gaussian Noise

Pufferfish Privacy Mechanisms for Correlated Data

Private Estimation with Public Data

Locally Private Gaussian Estimation

Privacy Guarantees in Posterior Sampling under Contamination

Improving Utility for Privacy-Preserving Analysis of Correlated Columns using Pufferfish Privacy

Optimal Noise-Adding Mechanism in Additive Differential Privacy

Pseudo Independent Conditional Approximation for Training the Mixtures of Gaussian Processes

Sample-Efficient Private Learning of Mixtures of Gaussians

Privately Learning Mixtures of Axis-Aligned Gaussians

Generalized Gaussian Mechanism for Differential Privacy

Better Gaussian Mechanism using Correlated Noise

Differential Privacy with Higher Utility by Exploiting Coordinate-wise Disparity: Laplace Mechanism Can Beat Gaussian in High Dimensions

General Gaussian Noise Mechanisms and Their Optimality for Unbiased Mean Estimation

Towards optimal noise distribution for privacy preserving in data aggregation

Mixed Laplace approximation for marginal posterior and Bayesian inference in error-in-operator model