Abstract:We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P(word|context): (1) No Language Structure in Context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns and models complex characteristics of language (e.g., syntax and semantics), while the TM simultaneously learns a latent representation from the entire document and discovers the underlying thematic structure. We unite two complementary paradigms of learning the meaning of word occurrences by combining a TM (e.g., DocNADE) and a LM in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited Context and/or Smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed DocNADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural LMs and embeddings priors that consistently outperform state-of-the-art generative TMs in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 6 long-text and 8 short-text datasets from diverse domains.

A Context-Aware Topic Model for Statistical Machine Translation.

Translation Model Adaptation for Statistical Machine Translation with Monolingual Topic Information.

Graph-Based Collective Lexical Selection for Statistical Machine Translation.

Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation

Adaptation of Language Models for SMT Using Neural Networks with Topic Information.

HanoiT: Enhancing Context-aware Translation via Selective Context

Measuring and Increasing Context Usage in Context-Aware Machine Translation

When Does Translation Require Context? A Data-driven, Multilingual Exploration

Statistical Word Sense Aware Topic Models

Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection

Towards Making the Most of Context in Neural Machine Translation

Contextual-LDA: A Context Coherent Latent Topic Model for Mining Large Corpora.

textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior

Improve Statistical Machine Translation with Context-Sensitive Bilingual Semantic Embedding Model.

Topic Models Incorporating Statistical Word Senses

Diving Deep into Context-Aware Neural Machine Translation

Attention Mechanism and Context Modeling System for Text Mining Machine Translation

Context-Adaptive Document-Level Neural Machine Translation

Topic-aware pivot language approach for statisticalmachine translation

InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling

Context-Aware Machine Translation with Source Coreference Explanation