Abstract:Supervised topic models simultaneously model the latent topic structure of large collections of documents and a response variable associated with each document. Existing inference methods are based on variational approximation or Monte Carlo sampling, which often suffers from the local minimum defect. Spectral methods have been applied to learn unsupervised topic models, such as latent Dirichlet allocation (LDA), with provable guarantees. This paper investigates the possibility of applying spectral methods to recover the parameters of supervised LDA (sLDA). We first present a two-stage spectral method, which recovers the parameters of LDA followed by a power update method to recover the regression model parameters. Then, we further present a single-phase spectral algorithm to jointly recover the topic distribution matrix as well as the regression weights. Our spectral algorithms are provably correct and computationally efficient. We prove a sample complexity bound for each algorithm and subsequently derive a sufficient condition for the identifiability of sLDA. Thorough experiments on synthetic and real-world datasets verify the theory and demonstrate the practical effectiveness of the spectral algorithms. In fact, our results on a large-scale review rating dataset demonstrate that our single-phase spectral algorithm alone gets comparable or even better performance than state-of-the-art methods, while previous work on spectral methods has rarely reported such promising performance.

Spectral Methods for Semi-Supervised Manifold Learning.

Spectral Transformation Approaches to Semi-supervised Learning

Geodesic Based Semi-supervised Multi-manifold Feature Extraction

Spectral Properties of the Alignment Matrices in Manifold Learning.

Maximum Margin Based Semi-supervised Spectral Kernel Learning.

A Regularized Approach for Geodesic-Based Semisupervised Multimanifold Learning

Semi-Supervised Discriminant Analysis Via Spectral Transduction.

The Hybrid of Semisupervised Manifold Learning and Spectrum Kernel for Classification

Diffusion-based Semi-supervised Spectral Algorithm for Regression on Manifolds

Semi-definite Manifold Alignment

Semi-Supervised Dimensionality Reduction of Hyperspectral Image Based on Sparse Multi-Manifold Learning

Spectral Clustering on Multiple Manifolds

Semi-supervised Local Multi-Manifold Isomap by Linear Embedding for Feature Extraction.

Manifold-based learning and synthesis.

Face Recognition Using Semi-Supervised Spectral Feature Selection

Locality Preserving Semi-Supervised Support Vector Machine.

Semisupervised Support Vector Machines with Tangent Space Intrinsic Manifold Regularization.

Spectral Learning for Supervised Topic Models

A Semi-supervised Manifold Alignment Algorithm and an Evaluation Method Based on Local Structure Preservation

Face Recognition Based on Semi-Supervised Manifold Learning

Active Semi-supervised Spectral Clustering