Abstract:In this paper we study the computation of the nonparametric maximum likelihood estimator (NPMLE) in multivariate mixture models. Our first approach discretizes this infinite dimensional convex optimization problem by fixing the support points of the NPMLE and optimizing over the mixture proportions. In this context we propose, leveraging the sparsity of the solution, an efficient and scalable semismooth Newton based augmented Lagrangian method (ALM). Our algorithm beats the state-of-the-art methods~\cite{koenker2017rebayes, kim2020fast} and can handle $n \approx 10^6$ data points with $m \approx 10^4$ support points. Our second procedure, which combines the expectation-maximization (EM) algorithm with the ALM approach above, allows for joint optimization of both the support points and the probability weights. For both our algorithms we provide formal results on their (superlinear) convergence properties. The computed NPMLE can be immediately used for denoising the observations in the framework of empirical Bayes. We propose new denoising estimands in this context along with their consistent estimates. Extensive numerical experiments are conducted to illustrate the effectiveness of our methods. In particular, we employ our procedures to analyze two astronomy data sets: (i) Gaia-TGAS Catalog~\cite{anderson2018improving} containing $n \approx 1.4 \times 10^6$ data points in two dimensions, and (ii) the $d=19$ dimensional data set from the APOGEE survey~\cite{majewski2017apache} with $n \approx 2.7 \times 10^4$.

Fast ML Estimation for the Mixture of Factor Analyzers Via an ECM Algorithm

Maximum likelihood estimation of factor analysis using the ECME algorithm with complete and incomplete data

Distributed Density Estimation Based on a Mixture of Factor Analyzers in a Sensor Network.

Maximum-Likelihood-Estimation Via the Ecm Algorithm - A General Framework

Maximum likelihood estimation via the ECM algorithm: A general framework

Bayesian mixtures of common factor analyzers: Model, variational inference, and applications

Large-scale gradient-based training of Mixtures of Factor Analyzers

The ECME Algorithm: A Simple Extension of EM and ECM with Faster Monotone Convergence

Model-based clustering via mixtures of unrestricted skew normal factor analyzers with complete and incomplete data

Semi-supervised Mixture of Latent Factor Analysis Models with Application to Online Key Variable Estimation

Maximum-likelihood Mixture Factor Analysis Model and Its Application for Process Monitoring

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Flexible High-Dimensional Unsupervised Learning with Missing Data

Splitting-while-merging framework for clustering high-dimension data with component-wise expectation conditional maximisation

A fast entropy assisted complete ensemble empirical mode decomposition algorithm

Recursive Mixture Factor Analyzer for Monitoring Multimode Time-Variant Industrial Processes

Robust estimation for mixtures of Gaussian factor analyzers, based on trimming and constraints

Competitive EM Algorithm for Finite Mixture Models

Infinite max-margin factor analysis via data augmentation

On Efficient and Scalable Computation of the Nonparametric Maximum Likelihood Estimator in Mixture Models