Abstract:The family of expectation--maximization (EM) algorithms provides a general approach to fitting flexible models for large and complex data. The expectation (E) step of EM-type algorithms is time-consuming in massive data applications because it requires multiple passes through the full data. We address this problem by proposing an asynchronous and distributed generalization of the EM called the distributed EM (DEM). Using DEM, existing EM-type algorithms are easily extended to massive data settings by exploiting the divide-and-conquer technique and widely available computing power, such as grid computing. The DEM algorithm reserves two groups of computing processes called workers and managers for performing the E step and the maximization step (M step), respectively. The samples are randomly partitioned into a large number of disjoint subsets and are stored on the worker processes. The E step of DEM algorithm is performed in parallel on all the workers, and every worker communicates its results to the managers at the end of local E step. The managers perform the M step after they have received results from a γ-fraction of the workers, where γ is a fixed constant in (0, 1]. The sequence of parameter estimates generated by the DEM algorithm retains the attractive properties of EM: convergence of the sequence of parameter estimates to a local mode and linear global rate of convergence. Across diverse simulations focused on linear mixed-effects models, the DEM algorithm is significantly faster than competing EM-type algorithms while having a similar accuracy. The DEM algorithm maintains its superior empirical performance on a movie ratings database consisting of 10 million ratings. Supplementary material for this article is available online.

STATISTICAL INFERENCE FOR MASSIVE DATA By

Distributed Statistical Inference for Massive Data

Distributed Bootstrap Simultaneous Inference for High-Dimensional Quantile Regression

A review of distributed statistical inference

A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniques

Optimal Subsampling Bootstrap for Massive Data

Estimation and testing of expectile regression with efficient subsampling for massive data

Distributed Computing and Inference for Big Data

On the asymptotic properties of a bagging estimator with a massive dataset

Selective Inference with Distributed Data

The Big Data Bootstrap

Dynamic statistical inference in massive datastreams

Communication-Efficient Accurate Statistical Estimation

Distributed Sufficient Dimension Reduction for Heterogeneous Massive Data

Optimal Distributed Subsampling for Maximum Quasi-Likelihood Estimators with Massive Data

Distributed Information Theoretic Clustering

An Asynchronous Distributed Expectation Maximization Algorithm for Massive Data: The DEM Algorithm

Efficient Estimation for Generalized Linear Models on a Distributed System with Nonrandomly Distributed Data

Distributed quantile regression for massive heterogeneous data

Statistical Methods and Computing for Big Data

Distributed Bayesian Matrix Decomposition for Big Data Mining and Clustering