Abstract:With the emergence and development of deep generative models, such as the variational auto-encoders (VAEs), the research on topic modeling successfully extends to a new area: neural topic modeling, which aims to learn disentangled topics to understand the data better. However, the original VAE framework had been shown to be limited in disentanglement performance, bringing their inherent defects to a neural topic model (NTM). In this paper, we put forward that the optimization objectives of contrastive learning are consistent with two important goals (alignment and uniformity) of well-disentangled topic learning. Also, the optimization objectives of contrastive learning are consistent with two key evaluation measures for topic models, topic coherence and topic diversity. So, we come to the important conclusion that alignment and uniformity of disentangled topic learning can be quantified with topic coherence and topic diversity. Accordingly, we are inspired to propose the Contrastive Disentangled Neural Topic Model (CNTM). By representing both words and topics as low-dimensional vectors in the same embedding space, we apply contrastive learning to neural topic modeling to produce factorized and disentangled topics in an interpretable manner. We compare our proposed CNTM with strong baseline models on widely-used metrics. Our model achieves the best topic coherence scores under the most general evaluation setting (100% proportion topic selected) with 25.0%, 10.9%, 24.6%, and 51.3% improvements above the second-best models’ scores reported on four datasets of 20 Newsgroups, Web Snippets, Tag My News, and Reuters, respectively. Our method also gets the second-best topic diversity scores on the dataset of 20Newsgroups and Web Snippets. Our experimental results show that CNTM can effectively leverage the disentanglement ability from contrastive learning to solve the inherent defect of neural topic modeling and obtain better topic quality.

A Novel Neural Topic Model and Its Supervised Extension.

A Novel Topic Model for Automatic Term Extraction

TAN-NTM: Topic Attention Networks for Neural Topic Modeling

Neural Topic Modeling with Deep Mutual Information Estimation

Topic Modeling Revisited: A Document Graph-based Neural Network Perspective

vONTSS: vMF based semi-supervised neural topic modeling with optimal transport

Enhancing Neural Topic Model with Multi-Level Supervisions from Seed Words

A survey on neural topic models: methods, applications, and challenges

Deep NMF topic modeling

Adaptation of Language Models for SMT Using Neural Networks with Topic Information.

Context Reinforced Neural Topic Modeling over Short Texts

S2vNTM: Semi-supervised vMF Neural Topic Modeling

Enhancing Topic Interpretability for Neural Topic Modeling Through Topic-Wise Contrastive Learning

A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings

Nonparametric Topic Modeling with Neural Inference

Improving neural topic modeling via Sinkhorn divergence

Neural Topic Modeling with Large Language Models in the Loop

Neural Topic Modeling with Cycle-Consistent Adversarial Training

Integration of Neural Embeddings and Probabilistic Models in Topic Modeling

Improving Topic Disentanglement Via Contrastive Learning