Abstract:Unsupervised learning aims at the discovery of hidden structure that drives the observations in the real world. It is essential for success in modern machine learning. Latent variable models are versatile in unsupervised learning and have applications in almost every domain. Training latent variable models is challenging due to the non-convexity of the likelihood objective. An alternative method is based on the spectral decomposition of low order moment tensors. This versatile framework is guaranteed to estimate the correct model consistently. My thesis spans both theoretical analysis of tensor decomposition framework and practical implementation of various applications. This thesis presents theoretical results on convergence to globally optimal solution of tensor decomposition using the stochastic gradient descent, despite non-convexity of the objective. This is the first work that gives global convergence guarantees for the stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points. This thesis also presents large-scale deployment of spectral methods carried out on various platforms. Dimensionality reduction techniques such as random projection are incorporated for a highly parallel and scalable tensor decomposition algorithm. We obtain a gain in both accuracies and in running times by several orders of magnitude compared to the state-of-art variational methods. To solve real world problems, more advanced models and learning algorithms are proposed. This thesis discusses generalization of LDA model to mixed membership stochastic block model for learning user communities in social network, convolutional dictionary model for learning word-sequence embeddings, hierarchical tensor decomposition and latent tree structure model for learning disease hierarchy, and spatial point process mixture model for detecting cell types in neuroscience.

NeuLFT: A Novel Approach to Nonlinear Canonical Polyadic Decomposition on High-Dimensional Incomplete Tensors

A Momentum-Incorporated Non-Negative Latent Factorization of Tensors Model for Dynamic Network Representation

Tensor Networks for Latent Variable Analysis: Higher Order Canonical Polyadic Decomposition

A Novel Approach to Large-Scale Dynamically Weighted Directed Network Representation

Adaptive Divergence-based Non-negative Latent Factor Analysis

A Biased Deep Tensor Factorization Network For Tensor Completion

Discovery of Latent Factors in High-dimensional Data Using Tensor Methods

DeepTensor: Low-Rank Tensor Decomposition with Deep Network Priors

Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach

HyperNTF: A Hypergraph Regularized Nonnegative Tensor Factorization for Dimensionality Reduction

A Multilayered-and-Randomized Latent Factor Model for High-Dimensional and Sparse Matrices

Multidimensional Data Processing With Bayesian Inference via Structural Block Decomposition

A Solution for Large Scale Nonlinear Regression with High Rank and Degree at Constant Memory Complexity via Latent Tensor Reconstruction

Latent Matrices for Tensor Network Decomposition and to Tensor Completion

Tensor Networks for Latent Variable Analysis. Part I: Algorithms for Tensor Train Decomposition

An Adaptive Latent Factorization of Tensors Model for Embedding Dynamic Communication Network

An Alternating-Direction-Method of Multipliers-Incorporated Approach to Symmetric Non-Negative Latent Factor Analysis

A Dynamic Linear Bias Incorporation Scheme for Nonnegative Latent Factor Analysis

Distributed Non-Negative Tensor Train Decomposition

Multi-constrained Symmetric Nonnegative Latent Factor Analysis for Accurately Representing Large-scale Undirected Weighted Networks

Latent Space Model for Higher-order Networks and Generalized Tensor Decomposition