Abstract:Expectation Maximization (EM) is among the most popular algorithms for maximum likelihood estimation, but it is generally only guaranteed to find its stationary points of the log-likelihood objective. The goal of this article is to present theoretical and empirical evidence that over-parameterization can help EM avoid spurious local optima in the log-likelihood. We consider the problem of estimating the mean vectors of a Gaussian mixture model in a scenario where the mixing weights are known. Our study shows that the global behavior of EM, when one uses an over-parameterized model in which the mixing weights are treated as unknown, is better than that when one uses the (correct) model with the mixing weights fixed to the known values. For symmetric Gaussians mixtures with two components, we prove that introducing the (statistically redundant) weight parameters enables EM to find the global maximizer of the log-likelihood starting from almost any initial mean parameters, whereas EM without this over-parameterization may very often fail. For other Gaussian mixtures, we provide empirical evidence that shows similar behavior. Our results corroborate the value of over-parameterization in solving non-convex optimization problems, previously observed in other domains.

On Convergence Properties of the EM Algorithm for Gaussian Mixtures.

On convergence properties of the EM algorithm for Gaussian mixtures

Convergence of Gradient EM on Multi-component Mixture of Gaussians.

Research on Correct Convergence of the EM Algorithm for Gaussian Mixtures

Convergence results for the EM approach to mixtures of experts architectures

Toward Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixture Models

On the Correct Convergence of the EM Algorithm for Gaussian Mixtures

Asymptotic Convergence Properties of the EM Algorithm with Respect to the Overlap in the Mixture

Comparative Analysis on Convergence Rates of the Em Algorithm and Its Two Modifications for Gaussian Mixtures

Statistical Convergence of the EM Algorithm on Gaussian Mixture Models

Asymptotic convergence rate of the EM algorithm for Gaussian mixtures

A gentle tutorial of the em algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

The EM Algorithm for Generalized Exponential Mixture Model

Benefits of over-parameterization with EM

An Efficient Em Approach To Parameter Learning Of The Mixture Of Gaussian Processes

EM algorithms of Gaussian mixture model and hidden Markov model

An Mcmc Based Em Algorithm For Mixtures Of Gaussian Processes

An Effective EM Algorithm for Mixtures of Gaussian Processes Via the MCMC Sampling and Approximation.

The Competitive EM Algorithm for Gaussian Mixtures with BYY Harmony Criterion

EM Algorithms for Gaussian Mixtures with Split-and-merge Operation