Abstract:We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P(word|context): (1) No Language Structure in Context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns and models complex characteristics of language (e.g., syntax and semantics), while the TM simultaneously learns a latent representation from the entire document and discovers the underlying thematic structure. We unite two complementary paradigms of learning the meaning of word occurrences by combining a TM (e.g., DocNADE) and a LM in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited Context and/or Smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed DocNADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural LMs and embeddings priors that consistently outperform state-of-the-art generative TMs in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 6 long-text and 8 short-text datasets from diverse domains.

System and methods for performing NLP related tasks using contextualized word representations

Incorporating Knowledge into Neural Network for Text Representation.

Deeper Text Understanding for IR with Contextual Neural Language Modeling

Contextual Word Representations: A Contextual Introduction

Hierarchical Contextualized Representation for Named Entity Recognition.

Explicit Contextual Semantics for Text Comprehension

Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection

Contextual LSTM (CLSTM) models for Large scale NLP tasks

Learning Semantic Representations for Novel Words: Leveraging Both Form and Context

Learning Context-Specific Word/Character Embeddings.

End-to-end Contextual Speech Recognition Using Class Language Models and a Token Passing Decoder.

Deep learning for language understanding of mental health concepts derived from Cognitive Behavioural Therapy

Training Word Embeddings for Deep Learning in Biomedical Text Mining Tasks

Probing Contextualized Sentence Representations with Visual Awareness

Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring

textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior

Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation

Dynamic Contextualized Word Embeddings

Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?

Lexicon Infused Phrase Embeddings for Named Entity Resolution

Broad Context Language Modeling As Reading Comprehension