Abstract:The family of expectation--maximization (EM) algorithms provides a general approach to fitting flexible models for large and complex data. The expectation (E) step of EM-type algorithms is time-consuming in massive data applications because it requires multiple passes through the full data. We address this problem by proposing an asynchronous and distributed generalization of the EM called the distributed EM (DEM). Using DEM, existing EM-type algorithms are easily extended to massive data settings by exploiting the divide-and-conquer technique and widely available computing power, such as grid computing. The DEM algorithm reserves two groups of computing processes called workers and managers for performing the E step and the maximization step (M step), respectively. The samples are randomly partitioned into a large number of disjoint subsets and are stored on the worker processes. The E step of DEM algorithm is performed in parallel on all the workers, and every worker communicates its results to the managers at the end of local E step. The managers perform the M step after they have received results from a γ-fraction of the workers, where γ is a fixed constant in (0, 1]. The sequence of parameter estimates generated by the DEM algorithm retains the attractive properties of EM: convergence of the sequence of parameter estimates to a local mode and linear global rate of convergence. Across diverse simulations focused on linear mixed-effects models, the DEM algorithm is significantly faster than competing EM-type algorithms while having a similar accuracy. The DEM algorithm maintains its superior empirical performance on a movie ratings database consisting of 10 million ratings. Supplementary material for this article is available online.

Acceleration of the EM algorithm using the vector epsilon algorithm

Acceleration of the EM Algorithm

Acceleration of the EM Algorithm Using the Vector Aitken Method and Its Steffensen Form

Damped Anderson acceleration with restarts and monotonicity control for accelerating EM and EM-like algorithms

The ECME Algorithm: A Simple Extension of EM and ECM with Faster Monotone Convergence

Parameter Expansion to Accelerate Em: the Px-Em Algorithm

Maximum-Likelihood-Estimation Via the Ecm Algorithm - A General Framework

Acceleration of the EM and ECM algorithms using the Aitken δ2 method for log-linear models with partially classified data

On the global and componentwise rates of convergence of the EM algorithm

Using Em to Obtain Asymptotic Variance - Covariance Matrices - the Sem Algorithm

Fast convergence of the Expectation Maximization algorithm under a logarithmic Sobolev inequality

Comparative Analysis on Convergence Rates of the Em Algorithm and Its Two Modifications for Gaussian Mixtures

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

A Mixed Evolutionary Algorithm to Solve the O-D Matrix Estimation Problem

The expectation-maximization algorithm for ill-posed integral equations: a convergence analysis

Regularized EM algorithm

Stochastic Expectation Maximization with Variance Reduction.

An Exact Line Search Scheme to Accelerate the EM Algorithm: Application to Gaussian Mixture Models Identification

An Asynchronous Distributed Expectation Maximization Algorithm for Massive Data: The DEM Algorithm

A fast entropy assisted complete ensemble empirical mode decomposition algorithm

Accelerated Algorithms for Eigen-Value Decomposition with Application to Spectral Clustering