Abstract:We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P(word|context): (1) No Language Structure in Context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns and models complex characteristics of language (e.g., syntax and semantics), while the TM simultaneously learns a latent representation from the entire document and discovers the underlying thematic structure. We unite two complementary paradigms of learning the meaning of word occurrences by combining a TM (e.g., DocNADE) and a LM in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited Context and/or Smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed DocNADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural LMs and embeddings priors that consistently outperform state-of-the-art generative TMs in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 6 long-text and 8 short-text datasets from diverse domains.

Context Reinforced Neural Topic Modeling over Short Texts

Short Text Understanding by Leveraging Knowledge into Topic Model.

A Novel Neural Topic Model and Its Supervised Extension.

Neural Topic Model with Distance Awareness

Let the Pretrained Language Models "Imagine" for Short Texts Topic Modeling

Utilizing Recurrent Neural Network for Topic Discovery in Short Text Scenarios

Robust Word-Network Topic Model For Short Texts

Topic Model over Short Texts Incorporating Word Embedding

Word Network Topic Model: A Simple but General Solution for Short and Imbalanced Texts

textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior

Neural Topic Model with Reinforcement Learning.

Topic Memory Networks for Short Text Classification

Topic model based on co-occurrence word networks for unbalanced short text datasets

Extracting Topics with SimultaneousWord Co-occurrence and Semantic Correlation Graphs: Neural Topic Modeling for Short Texts

Copula Guided Neural Topic Modelling for Short Texts

Incorporating Context-Relevant Knowledge into Convolutional Neural Networks for Short Text Classification

Topic Modeling Revisited: A Document Graph-based Neural Network Perspective

A survey on neural topic models: methods, applications, and challenges

Short Text Topic Model with Word Embeddings and Context Information

TAN-NTM: Topic Attention Networks for Neural Topic Modeling