Abstract:In this paper we study the computation of the nonparametric maximum likelihood estimator (NPMLE) in multivariate mixture models. Our first approach discretizes this infinite dimensional convex optimization problem by fixing the support points of the NPMLE and optimizing over the mixture proportions. In this context we propose, leveraging the sparsity of the solution, an efficient and scalable semismooth Newton based augmented Lagrangian method (ALM). Our algorithm beats the state-of-the-art methods~\cite{koenker2017rebayes, kim2020fast} and can handle $n \approx 10^6$ data points with $m \approx 10^4$ support points. Our second procedure, which combines the expectation-maximization (EM) algorithm with the ALM approach above, allows for joint optimization of both the support points and the probability weights. For both our algorithms we provide formal results on their (superlinear) convergence properties. The computed NPMLE can be immediately used for denoising the observations in the framework of empirical Bayes. We propose new denoising estimands in this context along with their consistent estimates. Extensive numerical experiments are conducted to illustrate the effectiveness of our methods. In particular, we employ our procedures to analyze two astronomy data sets: (i) Gaia-TGAS Catalog~\cite{anderson2018improving} containing $n \approx 1.4 \times 10^6$ data points in two dimensions, and (ii) the $d=19$ dimensional data set from the APOGEE survey~\cite{majewski2017apache} with $n \approx 2.7 \times 10^4$.

On Weight Design of Maximum Weighted Likelihood and an Extended EM Algorithm

Gaussian mixture density modeling and decomposition with weighted likelihood

Optimal Weighted Extreme Learning Machine for Imbalanced Learning with Differential Evolution [research Frontier]

Benefits of over-parameterization with EM

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Big Learning Expectation Maximization

Statistical Machine Learning EM algorithm Lecture Notes 13 : EM algorithm

Maximum Likelihood from Incomplete Data Via the EM Algorithm

Robust Fitting of Mixture Models using Weighted Complete Estimating Equations

Maximum-Likelihood-Estimation Via the Ecm Algorithm - A General Framework

On Convergence Properties of the EM Algorithm for Gaussian Mixtures.

Asymptotic Convergence Properties of Entropy Regularized Likelihood Learning on Finite Mixtures with Automatic Model Selection

TLS-EM Algorithm of Mixture Density Models for Exponential Families

A Novel Split and Merge EM Algorithm for Gaussian Mixture Model

Approaches in Using Expectation Maximization Algorithm for Maximum Likelihood Estimation of the Parameters of a Constrained State Space Model with an External Input Series

Dynamically regularized maximum likelihood learning of Gaussian mixtures

Using Maximum Weighted Likelihood to Derive Lehmer and Hölder Families of Means

Maximum Entropy Estimator for Hidden Markov Models: Reduction to Dimension 2

On Efficient and Scalable Computation of the Nonparametric Maximum Likelihood Estimator in Mixture Models

An Efficient Greedy EM Algorithm for Gaussian Mixture for Adaptive Model Selection Using the Kurtosis and Skewness Criterion

Maximum Likelihood Estimations and EM Algorithms with Length-biased Data