Abstract:We give a new algorithm for learning mixtures of $k$ Gaussians (with identity covariance in $\mathbb{R}^n$) to TV error $\varepsilon$, with quasi-polynomial ($O(n^{\text{poly log}\left(\frac{n+k}{\varepsilon}\right)})$) time and sample complexity, under a minimum weight assumption. Unlike previous approaches, most of which are algebraic in nature, our approach is analytic and relies on the framework of diffusion models. Diffusion models are a modern paradigm for generative modeling, which typically rely on learning the score function (gradient log-pdf) along a process transforming a pure noise distribution, in our case a Gaussian, to the data distribution. Despite their dazzling performance in tasks such as image generation, there are few end-to-end theoretical guarantees that they can efficiently learn nontrivial families of distributions; we give some of the first such guarantees. We proceed by deriving higher-order Gaussian noise sensitivity bounds for the score functions for a Gaussian mixture to show that that they can be inductively learned using piecewise polynomial regression (up to poly-logarithmic degree), and combine this with known convergence results for diffusion models. Our results extend to continuous mixtures of Gaussians where the mixing distribution is supported on a union of $k$ balls of constant radius. In particular, this applies to the case of Gaussian convolutions of distributions on low-dimensional manifolds, or more generally sets with small covering number.

Learning Probability Density Functions from Marginal Distributions with Applications to Gaussian Mixtures

Gaussian mixture density modeling and decomposition with weighted likelihood

Bayesian estimation and prediction for certain mixtures

Differentiability and Approximation of Probability Functions under Gaussian Mixture Models: A Bayesian Approach

A Specialized Probability Density Function for the Input of Mixture of Gaussian Processes.

Probabilistic Inference from Arbitrary Uncertainty using Mixtures of Factorized Generalized Gaussians

Recovering Risk-Neutral Probability Density Functions Using Gaussian Mixture Distributions

Learning Mixtures of Gaussians Using the DDPM Objective

A Generalized Finite Mixture Model for Asymmetric Probability Distribution Observations

Learning Mixtures of Gaussians Using Diffusion Models

Averaging, maximum penalized likelihood and Bayesian estimation for improving Gaussian mixture probability density estimates

Nearest Neighbor Dirichlet Mixtures

Probabilistic Classification by Density Estimation Using Gaussian Mixture Model and Masked Autoregressive Flow

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Gaussian Mixture Models with Rare Events.

PrivGMM: Probability Density Estimation with Local Differential Privacy

Entropic characterization of optimal rates for learning Gaussian mixtures

Self -Paced Mixture of T Distribution Model.

Gaussian Mixture Model with Rare Events

Mixture of Gaussian Processes and its Applications

Learning Gaussian Mixtures Using the Wasserstein-Fisher-Rao Gradient Flow