Abstract:Latent Variable Models (LVMs) are a large family of machine learning models providing a principled and effective way to extract underlying patterns, structure and knowledge from observed data. Due to the dramatic growth of volume and complexity of data, several new challenges have emerged and cannot be effectively addressed by existing LVMs: (1) How to capture long-tail patterns that carry crucial information when the popularity of patterns is distributed in a power-law fashion? (2) How to reduce model complexity and computational cost without compromising the modeling power of LVMs? (3) How to improve the interpretability and reduce the redundancy of discovered patterns? To addresses the three challenges discussed above, we develop a novel regularization technique for LVMs, which controls the geometry of the latent space during learning to enable the learned latent components of LVMs to be diverse in the sense that they are favored to be mutually different from each other, to accomplish long-tail coverage, low redundancy, and better interpretability. We propose a mutual angular regularizer (MAR) to encourage the components in LVMs to have larger mutual angles. The MAR is non-convex and non-smooth, entailing great challenges for optimization. To cope with this issue, we derive a smooth lower bound of the MAR and optimize the lower bound instead. We show that the monotonicity of the lower bound is closely aligned with the MAR to qualify the lower bound as a desirable surrogate of the MAR. Using neural network (NN) as an instance, we analyze how the MAR affects the generalization performance of NN. On two popular latent variable models --- restricted Boltzmann machine and distance metric learning, we demonstrate that MAR can effectively capture long-tail patterns, reduce model complexity without sacrificing expressivity and improve interpretability.

Development of Integrated Choice and Latent Variable (ICLV) Models Using Matrix-Based Analytic Approximation and Automatic Differentiation Methods on TensorFlow Platform

Analyzing and Improving Supervised Nonlinear Dynamical Probabilistic Latent Variable Model for Inferential Sensors

Discriminative conditional restricted Boltzmann machine for discrete choice and latent variable modelling

Learning in Integer Latent Variable Models with Nested Automatic Differentiation

Macrophage Migration Inhibitory Factor and Interleukin-8 Produced by Gastric Epithelial Cells during Helicobacter pylori Exposure Induce Expression and Activation of the Epidermal Growth Factor Receptor

BIVAS: A scalable Bayesian method for bi-level variable selection with applications

Diversity-Promoting Bayesian Learning of Latent Variable Models.

Variable Selection for High-dimensional Generalized Linear Models using an Iterated Conditional Modes/Medians Algorithm

A Multilayered-and-Randomized Latent Factor Model for High-Dimensional and Sparse Matrices

Kernel Bayesian logistic tensor decomposition with automatic rank determination for predicting multiple types of miRNA-disease associations

Latent Variable Modeling with Diversity-Inducing Mutual Angular Regularization

Maximum Ideal Likelihood Estimator: An New Estimation and Inference Framework for Latent Variable Models

TPLVM: Portfolio Construction by Student's $t$-process Latent Variable Model

Scalable Kernel Logistic Regression with Nyström Approximation: Theoretical Analysis and Application to Discrete Choice Modelling

Tensor Graphical Model: Non-Convex Optimization and Statistical Inference

New results on the asymptotic and finite sample properties of the MaCML approach to multinomial probit model estimation

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

Accelerating L1-penalized expectation maximization algorithm for latent variable selection in multidimensional two-parameter logistic models

A proportional-integral-derivative-incorporated stochastic gradient descent-based latent factor analysis model

Scalable Estimation of Multinomial Response Models with Random Consideration Sets

Oxygen Therapy Catheter, Mask, Hood and Tent