Abstract: This paper presents our recent efforts, zenLDA, an efficient and scalable Collapsed Gibbs Sampling system for Latent Dirichlet Allocation training, which is thought to be challenging that both data parallelism and model parallelism are required because of the Big sampling data with up to billions of documents and Big model size with up to trillions of parameters. zenLDA combines both algorithm level improvements and system level optimizations. It first presents a novel CGS algorithm that balances the time complexity, model accuracy and parallelization flexibility. The input corpus in zenLDA is represented as a directed graph and model parameters are annotated as the corresponding vertex attributes. The distributed training is parallelized by partitioning the graph that in each iteration it first applies CGS step for all partitions in parallel, followed by synchronizing the computed model each other. In this way, both data parallelism and model parallelism are achieved by converting them to graph parallelism. We revisited the tradeoff between system efficiency and model accuracy and presented approximations such as unsynchronized model, sparse model initialization and "converged" token exclusion. zenLDA is built on GraphX in Spark that provides distributed data abstraction (RDD) and expressive APIs to simplify the programming efforts and simultaneously hides the system complexities. This enables us to implement other CGS algorithm with a few lines of code change. To better fit in distributed data-parallel framework and achieve comparable performance with contemporary systems, we also presented several system level optimizations to push the performance limit. zenLDA was evaluated it against web-scale corpus, and the result indicates that zenLDA can achieve about much better performance than other CGS algorithm we implemented, and simultaneously achieve better model accuracy.

Distributing the Stochastic Gradient Sampler for Large-Scale LDA.

Federated Stochastic Gradient Langevin Dynamics

Large-scale and Scalable Latent Factor Analysis via Distributed Alternative Stochastic Gradient Descent for Recommender Systems

ZenLDA: An Efficient and Scalable Topic Model Training System on Distributed Data-Parallel Platform

Streaming Gibbs Sampling for LDA Model

PLDA+: Parallel latent dirichlet allocation with data placement and pipeline processing

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

ZenLDA: Large-scale Topic Model Training on Distributed Data-Parallel Platform

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

DisSAGD: A Distributed Parameter Update Scheme Based on Variance Reduction

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models

Sampling Sparse Representations with Randomized Measurement Langevin Dynamics

A Langevin-like Sampler for Discrete Distributions

CuLDA_CGS: Solving Large-scale LDA Problems on GPUs

Accelerating Topic Model Training on a Single Machine

Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework

Scaling up stochastic gradient descent for non-convex optimisation

Scaling Up Dynamic Topic Models

A generalization of regularized dual averaging and its dynamics

A Spectral Algorithm for Latent Dirichlet Allocation

Subsampling Error in Stochastic Gradient Langevin Diffusions