Abstract:Nested Chinese Restaurant Process (nCRP) topic models are powerful nonparametric Bayesian methods to extract a topic hierarchy from a given text corpus, where the hierarchical structure is automatically determined by the data. Hierarchical Latent Dirichlet Allocation (hLDA) is a popular instance of nCRP topic models. However, hLDA has only been evaluated at small scale, because the existing collapsed Gibbs sampling and instantiated weight variational inference algorithms either are not scalable or sacrifice inference quality with mean-field assumptions. Moreover, an efficient distributed implementation of the data structures, such as dynamically growing count matrices and trees, is challenging. In this paper, we propose a novel partially collapsed Gibbs sampling (PCGS) algorithm, which combines the advantages of collapsed and instantiated weight algorithms to achieve good scalability as well as high model quality. An initialization strategy is presented to further improve the model quality. Finally, we propose an efficient distributed implementation of PCGS through vectorization, pre-processing, and a careful design of the concurrent data structures and communication strategy. Empirical studies show that our algorithm is 111 times more efficient than the previous open-source implementation for hLDA, with comparable or even better model quality. Our distributed implementation can extract 1,722 topics from a 131-million-document corpus with 28 billion tokens, which is 4-5 orders of magnitude larger than the previous largest corpus, with 50 machines in 7 hours.

Hierarchical Topic Modeling with Nested Hierarchical Dirichlet Process

Interactive Topic Modeling Based on Hierarchical Dirichlet Process

Hierarchical Topic Mining via Joint Spherical Tree and Text Embedding

Incorporating Hierarchical Dirichlet Process into Tag Topic Model.

Learning Topic Hierarchies by Tree-Directed Latent Variable Models

Scalable Training of Hierarchical Topic Models

Hierarchical Dirichlet Processes with Social Influence

Hierarchical Online NMF for Detecting and Tracking Topic Hierarchies in a Text Stream

Hierarchical Clustering on HDP Topics to Build a Semantic Tree from Text.

Tracking and Connecting Topics Via Incremental Hierarchical Dirichlet Processes

A Knowledge-Based Semisupervised Hierarchical Online Topic Detection Framework.

Inferring Hierarchical Mixture Structures: A Bayesian Nonparametric Approach

Update Summarization using a Multi-level Hierarchical Dirichlet Process Model.

A Topic Model for Hierarchical Documents

An Improved Hierarchical Dirichlet Process-Hidden Markov Model and Its Application to Trajectory Modeling and Retrieval

Hierarchical Topic Integration Through Semi-Supervised Hierarchical Topic Modeling.

The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies

Scalable Inference for Nested Chinese Restaurant Process Topic Models

TSDPMM: Incorporating Prior Topic Knowledge into Dirichlet Process Mixture Models for Text Clustering.

Hierarchical Latent Semantic Mapping for Automated Topic Generation

Parallel dynamic topic modeling via evolving topic adjustment and term weighting scheme