Abstract:We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P(word|context): (1) No Language Structure in Context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns and models complex characteristics of language (e.g., syntax and semantics), while the TM simultaneously learns a latent representation from the entire document and discovers the underlying thematic structure. We unite two complementary paradigms of learning the meaning of word occurrences by combining a TM (e.g., DocNADE) and a LM in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited Context and/or Smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed DocNADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural LMs and embeddings priors that consistently outperform state-of-the-art generative TMs in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 6 long-text and 8 short-text datasets from diverse domains.

Deep Autoencoding Topic Model With Scalable Hybrid Bayesian Inference

WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling.

Max-Margin Deep Diverse Latent Dirichlet Allocation With Continual Learning

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

Bayesian Deep Embedding Topic Meta-Learner.

A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data

HVAE: A Deep Generative Model Via Hierarchical Variational Auto-Encoder for Multi-View Document Modeling

BAT-Chain: Bayesian-Aware Transport Chain for Topic Hierarchies Discovery

Bayesian Progressive Deep Topic Model with Knowledge Informed Textual Data Coarsening Process.

Deep Relational Topic Modeling Via Graph Poisson Gamma Belief Network.

Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network

Nonparametric Topic Modeling with Neural Inference

Layer-Assisted Neural Topic Modeling over Document Networks

Alleviating ``Posterior Collapse'' in Deep Topic Models via Policy Gradient

Scalable Training of Hierarchical Topic Models

A Bayesian Nonparametric Topic Model with Variational Auto-Encoders

textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior

Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings

Scalable Inference in Max-Margin Topic Models

Self-paced Compensatory Deep Boltzmann Machine for Semi-Structured Document Embedding

Neural Topic Modeling with Bidirectional Adversarial Training