Abstract:This paper studies the estimation of high-dimensional, discrete, possibly sparse, mixture models in topic models. The data consists of observed multinomial counts of $p$ words across $n$ independent documents. In topic models, the $p\times n$ expected word frequency matrix is assumed to be factorized as a $p\times K$ word-topic matrix $A$ and a $K\times n$ topic-document matrix $T$. Since columns of both matrices represent conditional probabilities belonging to probability simplices, columns of $A$ are viewed as $p$-dimensional mixture components that are common to all documents while columns of $T$ are viewed as the $K$-dimensional mixture weights that are document specific and are allowed to be sparse. The main interest is to provide sharp, finite sample, $\ell_1$-norm convergence rates for estimators of the mixture weights $T$ when $A$ is either known or unknown. For known $A$, we suggest MLE estimation of $T$. Our non-standard analysis of the MLE not only establishes its $\ell_1$ convergence rate, but reveals a remarkable property: the MLE, with no extra regularization, can be exactly sparse and contain the true zero pattern of $T$. We further show that the MLE is both minimax optimal and adaptive to the unknown sparsity in a large class of sparse topic distributions. When $A$ is unknown, we estimate $T$ by optimizing the likelihood function corresponding to a plug in, generic, estimator $\hat{A}$ of $A$. For any estimator $\hat{A}$ that satisfies carefully detailed conditions for proximity to $A$, the resulting estimator of $T$ is shown to retain the properties established for the MLE. The ambient dimensions $K$ and $p$ are allowed to grow with the sample sizes. Our application is to the estimation of 1-Wasserstein distances between document generating distributions. We propose, estimate and analyze new 1-Wasserstein distances between two probabilistic document representations.

Scalable Inference for Logistic-Normal Topic Models.

Scalable Inference in Max-Margin Topic Models

Improved Bayesian Logistic Supervised Topic Models with Data Augmentation.

Scaling Up Dynamic Topic Models

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

Scalable Estimation and Regularization for the Logistic Normal Multinomial Model

Scalable Inference for Nested Chinese Restaurant Process Topic Models

Latent Gaussian Models for Topic Modeling

Model-Parallel Inference for Big Topic Models

Likelihood estimation of sparse topic distributions in topic models and its applications to Wasserstein document distance calculations

Gibbs Max-margin Topic Models with Data Augmentation

Generalized Relational Topic Models with Data Augmentation.

Deep de Finetti: Recovering Topic Distributions from Large Language Models

Probabilistic Non-Negative Matrix Factorization and Its Robust Extensions for Topic Modeling.

Spectral Methods for Supervised Topic Models

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

A Nonparametric Model for Online Topic Discovery with Word Embeddings

Not Enough Data?: Joint Inferring Multiple Diffusion Networks via Network Generation Priors

Group Matrix Factorization for Scalable Topic Modeling

A 'Gibbs-Newton' Technique for Enhanced Inference of Multivariate Polya Parameters and Topic Models

Gibbs Max-Margin Topic Models with Fast Sampling Algorithms.