Abstract: This paper presents our recent efforts, zenLDA, an efficient and scalable Collapsed Gibbs Sampling system for Latent Dirichlet Allocation training, which is thought to be challenging that both data parallelism and model parallelism are required because of the Big sampling data with up to billions of documents and Big model size with up to trillions of parameters. zenLDA combines both algorithm level improvements and system level optimizations. It first presents a novel CGS algorithm that balances the time complexity, model accuracy and parallelization flexibility. The input corpus in zenLDA is represented as a directed graph and model parameters are annotated as the corresponding vertex attributes. The distributed training is parallelized by partitioning the graph that in each iteration it first applies CGS step for all partitions in parallel, followed by synchronizing the computed model each other. In this way, both data parallelism and model parallelism are achieved by converting them to graph parallelism. We revisited the tradeoff between system efficiency and model accuracy and presented approximations such as unsynchronized model, sparse model initialization and "converged" token exclusion. zenLDA is built on GraphX in Spark that provides distributed data abstraction (RDD) and expressive APIs to simplify the programming efforts and simultaneously hides the system complexities. This enables us to implement other CGS algorithm with a few lines of code change. To better fit in distributed data-parallel framework and achieve comparable performance with contemporary systems, we also presented several system level optimizations to push the performance limit. zenLDA was evaluated it against web-scale corpus, and the result indicates that zenLDA can achieve about much better performance than other CGS algorithm we implemented, and simultaneously achieve better model accuracy.

PLDA+: Parallel latent dirichlet allocation with data placement and pipeline processing

Parallel Topic Model and Its Application on Document Clustering.

GLDA: Parallel Gibbs Sampling for Latent Dirichlet Allocation on GPU.

LDPP: A Learned Directory Placement Policy in Distributed File Systems.

Distributing the Stochastic Gradient Sampler for Large-Scale LDA.

Parallel Inference for Latent Dirichlet Allocation on Graphics Processing Units.

HarpLDA+: Optimizing Latent Dirichlet Allocation for Parallel Efficiency

A More Time-Efficient Gibbs Sampling Algorithm Based on SparseLDA for Latent Dirichlet Allocation

CuLDA_CGS

CuLDA: Solving Large-scale LDA Problems on GPUs

CuLDA_CGS: Solving Large-scale LDA Problems on GPUs

A Comparative Study on Parallel Lda Algorithms in Mapreduce Framework

Communication-Efficient Parallel Belief Propagation for Latent Dirichlet Allocation

Gibbs Collapsed Sampling for Latent Dirichlet Allocation on Spark.

Analysis of large-scale distributed machine learning systems: a case study on LDA

Sparse Hybrid Variational-Gibbs Algorithm for Latent Dirichlet Allocation.

A Langevin-like Sampler for Discrete Distributions

Parallelization and Characterization of Probabilistic Latent Semantic Analysis

ZenLDA: An Efficient and Scalable Topic Model Training System on Distributed Data-Parallel Platform

High Performance LDA Through Collective Model Communication Optimization

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC