Abstract:The problem of topic modeling can be seen as a generalization of the clustering problem, in that it posits that observations are generated due to multiple latent factors (e.g., the words in each document are generated as a mixture of several active topics, as opposed to just one). This increased representational power comes at the cost of a more challenging unsupervised learning problem of estimating the topic probability vectors (the distributions over words for each topic), when only the words are observed and the corresponding topics are hidden. We provide a simple and efficient learning procedure that is guaranteed to recover the parameters for a wide class of mixture models, including the popular latent Dirichlet allocation (LDA) model. For LDA, the procedure correctly recovers both the topic probability vectors and the prior over the topics, using only trigram statistics (i.e., third order moments, which may be estimated with documents containing just three words). The method, termed Excess Correlation Analysis (ECA), is based on a spectral decomposition of low order moments (third and fourth order) via two singular value decompositions (SVDs). Moreover, the algorithm is scalable since the SVD operations are carried out on $k\times k$ matrices, where $k$ is the number of latent factors (e.g. the number of topics), rather than in the $d$-dimensional observed space (typically $d \gg k$).

Sparse Hybrid Variational-Gibbs Algorithm for Latent Dirichlet Allocation.

A Spectral Algorithm for Latent Dirichlet Allocation

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

Distributing the Stochastic Gradient Sampler for Large-Scale LDA.

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

Streaming Gibbs Sampling for LDA Model

Novel mixture allocation models for topic learning

Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling

Accelerating Topic Model Training on a Single Machine

A New Approach to Speeding Up Topic Modeling

A 'Gibbs-Newton' Technique for Enhanced Inference of Multivariate Polya Parameters and Topic Models

Optimize Collapsed Gibbs Sampling for Biterm Topic Model by Alias Method.

Incremental Variational Inference for Latent Dirichlet Allocation

A Fast Algorithm for Posterior Inference with Latent Dirichlet Allocation

Variable Selection for Latent Dirichlet Allocation

Scaling Up Dynamic Topic Models

Gaussian Hierarchical Latent Dirichlet Allocation: Bringing Polysemy Back

Efficient Methods for Incorporating Knowledge into Topic Models

Parallel Inference for Latent Dirichlet Allocation on Graphics Processing Units.

PLDA+: Parallel latent dirichlet allocation with data placement and pipeline processing

Algorithms of the LDA model [REPORT]