Abstract:The Expectation Maximisation (EM) algorithm is widely used to optimise non-convex likelihood functions with latent variables. Many authors modified its simple design to fit more specific situations. For instance, the Expectation (E) step has been replaced by Monte Carlo (MC), Markov Chain Monte Carlo or tempered approximations, etc. Most of the well-studied approximations belong to the stochastic class. By comparison, the literature is lacking when it comes to deterministic approximations. In this paper, we introduce a theoretical framework, with state-of-the-art convergence guarantees, for any deterministic approximation of the E step. We analyse theoretically and empirically several approximations that fit into this framework. First, for intractable E-steps, we introduce a deterministic version of MC-EM using Riemann sums. A straightforward method, not requiring any hyper-parameter fine-tuning, useful when the low dimensionality does not warrant a MC-EM. Then, we consider the tempered approximation, borrowed from the Simulated Annealing literature and used to escape local extrema. We prove that the tempered EM verifies the convergence guarantees for a wider range of temperature profiles than previously considered. We showcase empirically how new non-trivial profiles can more successfully escape adversarial initialisations. Finally, we combine the Riemann and tempered approximations into a method that accomplishes both their purposes.

Error bounds for computing the expectation by Markov chain Monte Carlo

Explicit error bounds for lazy reversible Markov Chain Monte Carlo

Discrepancy estimates for variance bounding Markov chain quasi-Monte Carlo

A non-asymptotic error analysis for parallel Monte Carlo estimation from many short Markov chains

Optimal approximating Markov chains for Bayesian inference

Marcinkiewicz's Strong Law of Large Numbers for Nonlinear Expectations

On Bounding and Approximating Functions of Multiple Expectations using Quasi-Monte Carlo

Convergence Speed and Approximation Accuracy of Numerical MCMC

A calculus for Markov chain Monte Carlo: studying approximations in algorithms

Statistical guarantees for stochastic Metropolis-Hastings

Stochastic Approximation for Expectation Objective and Expectation Inequality-Constrained Nonconvex Optimization

Learning from uniformly ergodic Markov chains

Asymptotic bias of inexact Markov Chain Monte Carlo methods in high dimension

Fast convergence of the Expectation Maximization algorithm under a logarithmic Sobolev inequality

Non-asymptotic Estimates for Markov Transition Matrices with Rigorous Error Bounds

Stochastic Approximation with Biased MCMC for Expectation Maximization

Markov Chain Variance Estimation: A Stochastic Approximation Approach

Unbiased Kinetic Langevin Monte Carlo with Inexact Gradients

Formal Error Bounds for the State Space Reduction of Markov Chains

On a Metropolis-Hastings importance sampling estimator

Deterministic Approximate EM Algorithm; Application to the Riemann Approximation EM and the Tempered EM