Abstract:The family of expectation--maximization (EM) algorithms provides a general approach to fitting flexible models for large and complex data. The expectation (E) step of EM-type algorithms is time-consuming in massive data applications because it requires multiple passes through the full data. We address this problem by proposing an asynchronous and distributed generalization of the EM called the distributed EM (DEM). Using DEM, existing EM-type algorithms are easily extended to massive data settings by exploiting the divide-and-conquer technique and widely available computing power, such as grid computing. The DEM algorithm reserves two groups of computing processes called workers and managers for performing the E step and the maximization step (M step), respectively. The samples are randomly partitioned into a large number of disjoint subsets and are stored on the worker processes. The E step of DEM algorithm is performed in parallel on all the workers, and every worker communicates its results to the managers at the end of local E step. The managers perform the M step after they have received results from a γ-fraction of the workers, where γ is a fixed constant in (0, 1]. The sequence of parameter estimates generated by the DEM algorithm retains the attractive properties of EM: convergence of the sequence of parameter estimates to a local mode and linear global rate of convergence. Across diverse simulations focused on linear mixed-effects models, the DEM algorithm is significantly faster than competing EM-type algorithms while having a similar accuracy. The DEM algorithm maintains its superior empirical performance on a movie ratings database consisting of 10 million ratings. Supplementary material for this article is available online.

Communication-efficient distributed EM algorithm

A communication efficient distributed one-step estimation

An Asynchronous Distributed Expectation Maximization Algorithm for Massive Data: The DEM Algorithm

Federated Expectation Maximization with heterogeneity mitigation and variance reduction

Distributed EM Algorithm for Gaussian Mixtures in Sensor Networks

Communication-Efficient Accurate Statistical Estimation

Optimal Sparsity-Sensitive Bounds for Distributed Mean Estimation.

Network EM Algorithm for Gaussian Mixture Model in Decentralized Federated Learning

Em Algorithm of the Truncated Multinormal Distribution with Linear Restriction on the Variables

Stochastic Expectation Maximization with Variance Reduction.

Heterogeneity-aware and communication-efficient distributed statistical inference

Robust Covariance Estimation for Distributed Principal Component Analysis

Improving Community Detection in Blockmodel by Distance-Based Observation Selection

DP-EM: Differentially Private Expectation Maximization

Distributed Sparse Recursive Least-Squares over Networks

Distributed Data Stream Clustering: A Fast EM-based Approach

Distributed Banach-Picard Iteration: Application to Distributed EM and Distributed PCA

On the global and componentwise rates of convergence of the EM algorithm

Communication-efficient distributed M-estimation with missing data

Communication-efficient Distributed Newton-like Optimization with Gradients and M-estimators

Distributed empirical risk minimization with differential privacy