Abstract:We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P(word|context): (1) No Language Structure in Context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns and models complex characteristics of language (e.g., syntax and semantics), while the TM simultaneously learns a latent representation from the entire document and discovers the underlying thematic structure. We unite two complementary paradigms of learning the meaning of word occurrences by combining a TM (e.g., DocNADE) and a LM in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited Context and/or Smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed DocNADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural LMs and embeddings priors that consistently outperform state-of-the-art generative TMs in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 6 long-text and 8 short-text datasets from diverse domains.

Explicit Word Density Estimation for Language Modelling

A neural probabilistic language model

OxLM: A Neural Language Modelling Framework for Machine Translation

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

Exploring the Limits of Language Modeling

Fast Oov Words Incorporation Using Structured Word Embeddings for Neural Network Language Model.

Neurons in Large Language Models: Dead, N-gram, Positional

Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction

Three new graphical models for statistical language modelling

textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior

Compressing Neural Language Models by Sparse Word Representations

Deep Neural Networks Language Model Based on CNN and LSTM Hybrid Architecture

Efficient Language Modeling with Sparse all-MLP

Exploiting Future Word Contexts in Neural Network Language Models for Speech Recognition.

Modelling Lexical Ambiguity with Density Matrices

Compositional Morphology for Word Representations and Language Modelling

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

Language models as master equation solvers

Exploring Extreme Quantization in Spiking Language Models

Enhancing Continuous Time Series Modelling with a Latent ODE-LSTM Approach

Efficient representation and fast look-up of Maximum Entropy language models.