Abstract:This paper studies the estimation of high-dimensional, discrete, possibly sparse, mixture models in topic models. The data consists of observed multinomial counts of $p$ words across $n$ independent documents. In topic models, the $p\times n$ expected word frequency matrix is assumed to be factorized as a $p\times K$ word-topic matrix $A$ and a $K\times n$ topic-document matrix $T$. Since columns of both matrices represent conditional probabilities belonging to probability simplices, columns of $A$ are viewed as $p$-dimensional mixture components that are common to all documents while columns of $T$ are viewed as the $K$-dimensional mixture weights that are document specific and are allowed to be sparse. The main interest is to provide sharp, finite sample, $\ell_1$-norm convergence rates for estimators of the mixture weights $T$ when $A$ is either known or unknown. For known $A$, we suggest MLE estimation of $T$. Our non-standard analysis of the MLE not only establishes its $\ell_1$ convergence rate, but reveals a remarkable property: the MLE, with no extra regularization, can be exactly sparse and contain the true zero pattern of $T$. We further show that the MLE is both minimax optimal and adaptive to the unknown sparsity in a large class of sparse topic distributions. When $A$ is unknown, we estimate $T$ by optimizing the likelihood function corresponding to a plug in, generic, estimator $\hat{A}$ of $A$. For any estimator $\hat{A}$ that satisfies carefully detailed conditions for proximity to $A$, the resulting estimator of $T$ is shown to retain the properties established for the MLE. The ambient dimensions $K$ and $p$ are allowed to grow with the sample sizes. Our application is to the estimation of 1-Wasserstein distances between document generating distributions. We propose, estimate and analyze new 1-Wasserstein distances between two probabilistic document representations.

Sparse Poisson coding for high dimensional document clustering

Document Clustering Using Locality Preserving Indexing

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

A Practical Algorithm for Solving the Sparseness Problem of Short Text Clustering

A parsimonious family of multivariate Poisson-lognormal distributions for clustering multivariate count data

Document clustering based on nonnegative sparse matrix factorization

Document clustering using locality preserving indexing and support vector machines

Data-Dependent Sparsity for Subspace Clustering.

Tensor LRR and Sparse Coding-Based Subspace Clustering.

Sparse Subspace Clustering Using Square-Root Penalty

Low-Rank Graph Regularized Sparse Coding

Adaptive Lasso and group-Lasso for functional Poisson regression

Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization

Extended Strategies For Document Clustering With Word Co-Occurrences

A Low-Dimensional Representation Learning Method for Text Classification and Clustering

Subspace Clustering of Very Sparse High-Dimensional Data

Learning an Optimal Bipartite Graph for Subspace Clustering Via Constrained Laplacian Rank

Sparse Subspace Clustering via Diffusion Process

Sparse Poisson Regression with Penalized Weighted Score Function

$\Ell_p$ Norm Independently Interpretable Regularization Based Sparse Coding for Highly Correlated Data

Likelihood estimation of sparse topic distributions in topic models and its applications to Wasserstein document distance calculations