Abstract:We consider the question of estimating multi-dimensional Gaussian mixtures (GM) with compactly supported or subgaussian mixing distributions. Minimax estimation rate for this class (under Hellinger, TV and KL divergences) is a long-standing open question, even in one dimension. In this paper we characterize this rate (for all constant dimensions) in terms of the metric entropy of the class. Such characterizations originate from seminal works of Le Cam (1973); Birge (1983); Haussler and Opper (1997); Yang and Barron (1999). However, for GMs a key ingredient missing from earlier work (and widely sought-after) is a comparison result showing that the KL and the squared Hellinger distance are within a constant multiple of each other uniformly over the class. Our main technical contribution is in showing this fact, from which we derive entropy characterization for estimation rate under Hellinger and KL. Interestingly, the sequential (online learning) estimation rate is characterized by the global entropy, while the single-step (batch) rate corresponds to local entropy, paralleling a similar result for the Gaussian sequence model recently discovered by Neykov (2022) and Mourtada (2023). Additionally, since Hellinger is a proper metric, our comparison shows that GMs under KL satisfy the triangle inequality within multiplicative constants, implying that proper and improper estimation rates coincide.

A Fast Globally Supervised Learning Algorithm for Gaussian Mixture Models

Gaussian mixture density modeling and decomposition with weighted likelihood

A revision for gaussian mixture density decomposition algorithm

A Greedy Merge Learning Algorithm for Gaussian Mixture Model

Unsupervised Classification Based on Penalized Maximum Likelihood of Gaussian Mixture Models

A Robust Infinite Gaussian Mixture Model and Its Application in Fault Detection of Nonlinear Multimode Processes

Renyi entropy penalized learning algorithm for Gaussian mixture with automated model selection

The Supervised Learning Gaussian Mixture Model

Dynamically regularized maximum likelihood learning of Gaussian mixtures

Entropic characterization of optimal rates for learning Gaussian mixtures

Learning Gaussian Mixtures Using the Wasserstein-Fisher-Rao Gradient Flow

Learning Gaussian mixture model with a maximization-maximization algorithm for image classification

Entropy Penalized Automated Model Selection On Gaussian Mixture

A gradient entropy regularized likelihood learning algorithm on gaussian mixture with automatic model selection

A Semi-Supervised Learning Algorithm on Gaussian Mixture with Automatic Model Selection

Learning Gaussian Mixture Models By Structural Risk Minimization

Gaussian Mixture Models with Rare Events.

A Fast Fixed-Point Byy Harmony Learning Algorithm on Gaussian Mixture with Automated Model Selection

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Gaussian Mixture Model with Rare Events

Gaussian Mixture Learning Via Robust Competitive Agglomeration