Abstract:We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P(word|context): (1) No Language Structure in Context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns and models complex characteristics of language (e.g., syntax and semantics), while the TM simultaneously learns a latent representation from the entire document and discovers the underlying thematic structure. We unite two complementary paradigms of learning the meaning of word occurrences by combining a TM (e.g., DocNADE) and a LM in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited Context and/or Smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed DocNADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural LMs and embeddings priors that consistently outperform state-of-the-art generative TMs in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 6 long-text and 8 short-text datasets from diverse domains.

Integration of Neural Embeddings and Probabilistic Models in Topic Modeling

Incorporating Knowledge Graph Embeddings into Topic Modeling

Incorporating Probabilistic Knowledge into Topic Models.

Mining Coherent Topics in Documents Using Word Embeddings and Large-Scale Text Data

Hierarchical Topic Modeling with Nested Hierarchical Dirichlet Process

A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings

Nonparametric Topic Modeling with Neural Inference

Topic Modeling Using Distributed Word Embeddings

Integrating Topic Modeling with Word Embeddings by Mixtures of Vmfs.

Hierarchical Neural Topic Model with Embedding Cluster and Neural Variational Inference.

Neural Topic Model with Distance Awareness

Topic Modeling Revisited: A Document Graph-based Neural Network Perspective

Neural Topic Modeling with Deep Mutual Information Estimation

Probabilistic Topic Modelling with Transformer Representations

Knowledge-Aware Bayesian Deep Topic Model

Neural Variational Gaussian Mixture Topic Model.

Topic Modeling over Short Texts by Incorporating Word Embeddings

textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior

A Novel Neural Topic Model and Its Supervised Extension.

Extracting nonlinear neural topics with neural variational bayes

Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings