Abstract:Data augmentation (DA) algorithms are widely used for Bayesian inference due to their simplicity. In massive data settings, however, DA algorithms are prohibitively slow because they pass through the full data in any iteration, imposing serious restrictions on their usage despite the advantages. Addressing this problem, we develop a framework for extending any DA that exploits asynchronous and distributed computing. The extended DA algorithm is indexed by a parameter $r \in (0, 1)$ and is called Asynchronous and Distributed (AD) DA with the original DA as its parent. Any ADDA starts by dividing the full data into $k$ smaller disjoint subsets and storing them on $k$ processes, which could be machines or processors. Every iteration of ADDA augments only an $r$-fraction of the $k$ data subsets with some positive probability and leaves the remaining $(1-r)$-fraction of the augmented data unchanged. The parameter draws are obtained using the $r$-fraction of new and $(1-r)$-fraction of old augmented data. For many choices of $k$ and $r$, the fractional updates of ADDA lead to a significant speed-up over the parent DA in massive data settings, and it reduces to the distributed version of its parent DA when $r=1$. We show that the ADDA Markov chain is Harris ergodic with the desired stationary distribution under mild conditions on the parent DA algorithm. We demonstrate the numerical advantages of the ADDA in three representative examples corresponding to different kinds of massive data settings encountered in applications. In all these examples, our DA generalization is significantly faster than its parent DA algorithm for all the choices of $k$ and $r$. We also establish geometric ergodicity of the ADDA Markov chain for all three examples, which in turn yields asymptotically valid standard errors for estimates of desired posterior quantities.

Sample-Based Attribute Selective A$n$ DE for Large Data

Selective AnDE for Large Data Learning: a Low-Bias Memory Constrained Approach

Learning by Extrapolation from Marginal to Full-Multivariate Probability Distributions: Decreasingly Naive Bayesian Classification

MiniAnDE: a reduced AnDE ensemble to deal with microarray data

Alleviating the Attribute Conditional Independence and I.I.D. Assumptions of Averaged One-Dependence Estimator by Double Weighting

Averaged Tree-Augmented One-Dependence Estimators

Big Models for Big Data using Multi objective averaged one dependence estimators

Selective AnDE Based on Attributes Ranking by Maximin Conditional Mutual Information (MMCMI)

Self-Adaptive Attribute Value Weighting for Averaged One-Dependence Estimators.

Multi-loop Adaptive Differential Evolution for Large-Scale Expensive Optimization

Accelerating large-scale DEA computation using sequential categorization and dynamic reference set selection

Efficient Data-aware Distance Comparison Operations for High-Dimensional Approximate Nearest Neighbor Search

Extracting Credible Dependencies for Averaged One-Dependence Estimator Analysis

Adaptive Feature Selection With Augmented Attributes

Asynchronous and Distributed Data Augmentation for Massive Data Settings

K-Dependence Bayesian Classifier Ensemble

Scalable Bayesian regression in high dimensions with multiple data sources

Scalable Inference for Massive Data

Attribute Value Weighted Average of One-Dependence Estimators

To Select or to Weigh: A Comparative Study of Linear Combination Schemes for SuperParent-One-Dependence Estimators

AdaSelection: Accelerating Deep Learning Training through Data Subsampling