Abstract:Finite mixture models are among the most popular statistical models used in different data science disciplines. Despite their broad applicability, inference under these models typically leads to computationally challenging non-convex problems. While the Expectation-Maximization (EM) algorithm is the most popular approach for solving these non-convex problems, the behavior of this algorithm is not well understood. In this work, we focus on the case of mixture of Laplacian (or Gaussian) distribution. We start by analyzing a simple equally weighted mixture of two single dimensional Laplacian distributions and show that every local optimum of the population maximum likelihood estimation problem is globally optimal. Then, we prove that the EM algorithm converges to the ground truth parameters almost surely with random initialization. Our result extends the existing results for Gaussian distribution to Laplacian distribution. Then we numerically study the behavior of mixture models with more than two components. Motivated by our extensive numerical experiments, we propose a novel stochastic method for estimating the mean of components of a mixture model. Our numerical experiments show that our algorithm outperforms the Naive EM algorithm in almost all scenarios.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the behavioral analysis of the Expectation - Maximization (EM) algorithm in the mixture model, especially for the mixture model of Laplace distribution (or Gaussian distribution). Specifically, the paper focuses on the following aspects: 1. **Analysis of global optimality in simple cases**: The paper first analyzes a simple mixture model of two one - dimensional Laplace distributions with equal weights and proves that every local optimal solution in the maximum - likelihood estimation problem of this model is a global optimal solution. 2. **Convergence of the EM algorithm**: Next, the paper proves that in the case of random initialization, the EM algorithm converges to the true parameters almost surely. This result extends the research on the Gaussian distribution mixture model in the existing literature and applies it to the Laplace distribution mixture model. 3. **Numerical study of multi - component mixture models**: Based on extensive numerical experiments, the paper further studies the behavior of mixture models containing more than two components and proposes a new stochastic method to estimate the means of each component of the mixture model. The experimental results show that the newly proposed algorithm outperforms the traditional naive EM algorithm in almost all cases. 4. **Improved EM algorithm**: Aiming at the problem of multiple local optimal solutions in the mixture model, the paper proposes an improved EM algorithm. By introducing a regularization term to enforce the first - order moment condition, the performance of the algorithm is improved. In addition, an EM algorithm based on stochastic multi - objective optimization is also proposed, which further increases the probability of converging to the true parameters from random initial points. In summary, this paper aims to deeply understand the behavior of the EM algorithm when dealing with mixture models and provides an improved method through theoretical analysis and numerical experiments to improve the performance of the EM algorithm in practical applications.

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Gaussian mixture density modeling and decomposition with weighted likelihood

Model-based clustering and classification using mixtures of multivariate skewed power exponential distributions

Statistical Convergence of the EM Algorithm on Gaussian Mixture Models

Local maxima in the likelihood of gaussian mixture models: Structural results and algorithmic consequences

Benefits of over-parameterization with EM

The EM Algorithm for Generalized Exponential Mixture Model

An Effective EM Algorithm for Mixtures of Gaussian Processes Via the MCMC Sampling and Approximation.

Convergence of Gradient EM on Multi-component Mixture of Gaussians.

On Efficient and Scalable Computation of the Nonparametric Maximum Likelihood Estimator in Mixture Models

Big Learning Expectation Maximization

Convergence results for the EM approach to mixtures of experts architectures

On convergence properties of the EM algorithm for Gaussian mixtures

Parameter Estimation for Gaussian Mixture Processes Based on Expectation-Maximization Method

A Novel Split and Merge EM Algorithm for Gaussian Mixture Model

A gentle tutorial of the em algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models

An Efficient Em Approach To Parameter Learning Of The Mixture Of Gaussian Processes

Asymptotic Convergence Properties of the EM Algorithm with Respect to the Overlap in the Mixture

An Improved Mixture Model of Gaussian Processes and Its Classification Expectation–Maximization Algorithm

On Convergence Properties of the EM Algorithm for Gaussian Mixtures.

An Mcmc Based Em Algorithm For Mixtures Of Gaussian Processes