A Stochastic Multivariate Latent Variable Model For Categorical Responses

Mahdi Mollakazemiha

2023-06-02

Abstract:This paper introduces a mathematical framework of a stochastic process model as a generalization of diffusion stochastic processes to model latent variables in categorical responses given unobserved random effects and maximum likelihood estimation of parameters is indicated.

Statistics Theory

What problem does this paper attempt to address?

Some Remarks on Latent Variable Models in Categorical Data Analysis

Alan Agresti,Maria Kateri

DOI: https://doi.org/10.1080/03610926.2013.814783

2014-01-27

Abstract:We present an overview of some important and/or interesting contributions to the latent variable literature for the analysis of multivariate categorical responses, beginning with Lazarsfeld's introduction of latent class models. There is by now an enormous literature on latent variable models for categorical responses, especially in the context of including random effects in generalized linear mixed models, so this is necessarily a highly selective overview. Due to space considerations, we summarize the main ideas, suppressing details. As part of our presentation, we raise a couple of questions that may suggest future research work.
A Latent-Variable Bayesian Nonparametric Regression Model

George Karabatsos,Stephen G. Walker

DOI: https://doi.org/10.48550/arXiv.1212.3712

2013-01-03

Abstract:We introduce a random partition model for Bayesian nonparametric regression. The model is based on infinitely-many disjoint regions of the range of a latent covariate-dependent Gaussian process. Given a realization of the process, the cluster of dependent variable responses that share a common region are assumed to arise from the same distribution. Also, the latent Gaussian process prior allows for the random partitions (i.e., clusters of the observations) to exhibit dependencies among one another. The model is illustrated through the analysis of a real data set arising from education, and through the analysis of simulated data that were generated from complex data-generating models.

Methodology
Stochastic Differential Equations with Variational Wishart Diffusions

Martin Jørgensen,Marc Peter Deisenroth,Hugh Salimbeni

DOI: https://doi.org/10.48550/arXiv.2006.14895

2020-06-26

Abstract:We present a Bayesian non-parametric way of inferring stochastic differential equations for both regression tasks and continuous-time dynamical modelling. The work has high emphasis on the stochastic part of the differential equation, also known as the diffusion, and modelling it by means of Wishart processes. Further, we present a semi-parametric approach that allows the framework to scale to high dimensions. This successfully lead us onto how to model both latent and auto-regressive temporal systems with conditional heteroskedastic noise. We provide experimental evidence that modelling diffusion often improves performance and that this randomness in the differential equation can be essential to avoid overfitting.

Machine Learning
Modeling Multivariate Mixed-Response Functional Data

Beth A. Tidemann-Miller,Brian J. Reich,Ana-Maria Staicu

DOI: https://doi.org/10.48550/arXiv.1601.02461

2016-01-11

Abstract:We propose a Bayesian modeling framework for jointly analyzing multiple functional responses of different types (e.g. binary and continuous data). Our approach is based on a multivariate latent Gaussian process and models the dependence among the functional responses through the dependence of the latent process. Our framework easily accommodates additional covariates. We offer a way to estimate the multivariate latent covariance, allowing for implementation of multivariate functional principal components analysis (FPCA) to specify basis expansions and simplify computation. We demonstrate our method through both simulation studies and an application to real data from a periodontal study.

Methodology
Bayesian Analysis of Binary and Polychotomous Response Data

James H. Albert,Siddhartha Chib

DOI: https://doi.org/10.1080/01621459.1993.10476321

IF: 4.369

1993-06-01

Journal of the American Statistical Association

Abstract:A vast literature in statistics, biometrics, and econometrics is concerned with the analysis of binary and polychotomous response data. The classical approach fits a categorical response regression model using maximum likelihood, and inferences about the model are based on the associated asymptotic theory. The accuracy of classical confidence statements is questionable for small sample sizes. In this article, exact Bayesian methods for modeling categorical response data are developed using the idea of data augmentation. The general approach can be summarized as follows. The probit regression model for binary outcomes is seen to have an underlying normal regression structure on latent continuous data. Values of the latent data can be simulated from suitable truncated normal distributions. If the latent data are known, then the posterior distribution of the parameters can be computed using standard results for normal linear models. Draws from this posterior are used to sample new latent data, and the process is iterated with Gibbs sampling. This data augmentation approach provides a general framework for analyzing binary regression models. It leads to the same simplification achieved earlier for censored regression models. Under the proposed framework, the class of probit regression models can be enlarged by using mixtures of normal distributions to model the latent data. In this normal mixture class, one can investigate the sensitivity of the parameter estimates to the choice of “link function,” which relates the linear regression estimate to the fitted probabilities. In addition, this approach allows one to easily fit Bayesian hierarchical models. One specific model considered here reflects the belief that the vector of regression coefficients lies on a smaller dimension linear subspace. The methods can also be generalized to multinomial response models with J > 2 categories. In the ordered multinomial model, the J categories are ordered and a model is written linking the cumulative response probabilities with the linear regression structure. In the unordered multinomial model, the latent variables have a multivariate normal distribution with unknown variance-covariance matrix. For both multinomial models, the data augmentation method combined with Gibbs sampling is outlined. This approach is especially attractive for the multivariate probit model, where calculating the likelihood can be difficult.

statistics & probability
A Fused Latent and Graphical Model for Multivariate Binary Data

Yunxiao Chen,Xiaoou Li,Jingchen Liu,Zhiliang Ying

DOI: https://doi.org/10.48550/arXiv.1606.08925

2016-06-29

Abstract:We consider modeling, inference, and computation for analyzing multivariate binary data. We propose a new model that consists of a low dimensional latent variable component and a sparse graphical component. Our study is motivated by analysis of item response data in cognitive assessment and has applications to many disciplines where item response data are collected. Standard approaches to item response data in cognitive assessment adopt the multidimensional item response theory (IRT) models. However, human cognition is typically a complicated process and thus may not be adequately described by just a few factors. Consequently, a low-dimensional latent factor model, such as the multidimensional IRT models, is often insufficient to capture the structure of the data. The proposed model adds a sparse graphical component that captures the remaining ad hoc dependence. It reduces to a multidimensional IRT model when the graphical component becomes degenerate. Model selection and parameter estimation are carried out simultaneously through construction of a pseudo-likelihood function and properly chosen penalty terms. The convexity of the pseudo-likelihood function allows us to develop an efficient algorithm, while the penalty terms generate a low-dimensional latent component and a sparse graphical structure. Desirable theoretical properties are established under suitable regularity conditions. The method is applied to the revised Eysenck's personality questionnaire, revealing its usefulness in item analysis. Simulation results are reported that show the new method works well in practical situations.

Methodology
Bayesian Semiparametric Longitudinal Inverse-Probit Mixed Models for Category Learning

Minerva Mukhopadhyay,Jacie R. McHaney,Bharath Chandrasekaran,Abhra Sarkar

DOI: https://doi.org/10.1007/s11336-024-09947-8

2024-02-21

Psychometrika

Abstract:Understanding how the adult human brain learns novel categories is an important problem in neuroscience. Drift-diffusion models are popular in such contexts for their ability to mimic the underlying neural mechanisms. One such model for gradual longitudinal learning was recently developed in Paulon et al. (J Am Stat Assoc 116:1114–1127, 2021). In practice, category response accuracies are often the only reliable measure recorded by behavioral scientists to describe human learning. Category response accuracies are, however, often the only reliable measure recorded by behavioral scientists to describe human learning. To our knowledge, however, drift-diffusion models for such scenarios have never been considered in the literature before. To address this gap, in this article, we build carefully on Paulon et al. (J Am Stat Assoc 116:1114–1127, 2021), but now with latent response times integrated out, to derive a novel biologically interpretable class of 'inverse-probit' categorical probability models for observed categories alone. However, this new marginal model presents significant identifiability and inferential challenges not encountered originally for the joint model in Paulon et al. (J Am Stat Assoc 116:1114–1127, 2021). We address these new challenges using a novel projection-based approach with a symmetry-preserving identifiability constraint that allows us to work with conjugate priors in an unconstrained space. We adapt the model for group and individual-level inference in longitudinal settings. Building again on the model's latent variable representation, we design an efficient Markov chain Monte Carlo algorithm for posterior computation. We evaluate the empirical performance of the method through simulation experiments. The practical efficacy of the method is illustrated in applications to longitudinal tone learning studies.

psychology, mathematical,mathematics, interdisciplinary applications,social sciences, mathematical methods
Multivariate Matérn Models -- A Spectral Approach

Drew Yarger,Stilian Stoev,Tailen Hsing

2024-06-02

Abstract:The classical Matérn model has been a staple in spatial statistics. Novel data-rich applications in environmental and physical sciences, however, call for new, flexible vector-valued spatial and space-time models. Therefore, the extension of the classical Matérn model has been a problem of active theoretical and methodological interest. In this paper, we offer a new perspective to extending the Matérn covariance model to the vector-valued setting. We adopt a spectral, stochastic integral approach, which allows us to address challenging issues on the validity of the covariance structure and at the same time to obtain new, flexible, and interpretable models. In particular, our multivariate extensions of the Matérn model allow for asymmetric covariance structures. Moreover, the spectral approach provides an essentially complete flexibility in modeling the local structure of the process. We establish closed-form representations of the cross-covariances when available, compare them with existing models, simulate Gaussian instances of these new processes, and demonstrate estimation of the model's parameters through maximum likelihood. An application of the new class of multivariate Matérn models to environmental data indicate their success in capturing inherent covariance-asymmetry phenomena.

Methodology,Statistics Theory,Other Statistics
Generative Classification Model for Categorical Data Based on Latent Gaussian Process.

Fengmao Lv,Guowu Yang,William Zhu,Chuan Liu

DOI: https://doi.org/10.1016/j.patrec.2017.03.025

IF: 4.757

2017-01-01

Pattern Recognition Letters

Abstract:In many machine learning applications such as computer-aided diagnosis, gene sequence analysis or natural language processing, categorical data appears. For small-scale data set with high dimensions, since relatively small proportion of possible categorical configurations are covered by training samples, conventional methods based on frequency information such as Dirichlet Compound Multinomial distribution usually runs into problems of over-fitting. Latent gaussian process is an effective bayesian non-parametric technique for categorical data modeling, which was proposed as an unsupervised method to embed unlabelled categorical data into a continuous and low-dimensional space through gaussian process. As a probabilistic generative model, latent gaussian process owns the ability of density estimation. In this paper, we propose a generative classification model as a supervised method for labelled categorical data, in which we use latent gaussian process to estimate the class-conditional densities. Since the complexity of gaussian process model can adapt to the size of training data, our method is able to effectively model small-sale categorical data. Experimental results show that our proposal can achieve better classification performance compared with other classification models for categorical data. (C) 2017 Elsevier B.V. All rights reserved.
A flexible Bayesian generalized linear model for dichotomous response data with an application to text categorization

Susana Eyheramendy,David Madigan

DOI: https://doi.org/10.48550/arXiv.0708.0959

2007-08-07

Methodology

Abstract:We present a class of sparse generalized linear models that include probit and logistic regression as special cases and offer some extra flexibility. We provide an EM algorithm for learning the parameters of these models from data. We apply our method in text classification and in simulated data and show that our method outperforms the logistic and probit models and also the elastic net, in general by a substantial margin.
A Latent Topic Model with Markov Transition for Process Data

Haochen Xu,Guanhua Fang,Zhiliang Ying

DOI: https://doi.org/10.1111/bmsp.12197

2019-01-01

Abstract:We propose a latent topic model with a Markov transition for process data, which consists of time-stamped events recorded in a log file. Such data are becoming more widely available in computer-based educational assessment with complex problem-solving items. The proposed model can be viewed as an extension of the hierarchical Bayesian topic model with a hidden Markov structure to accommodate the underlying evolution of an examinee's latent state. Using topic transition probabilities along with response times enables us to capture examinees' learning trajectories, making clustering/classification more efficient. A forward-backward variational expectation-maximization (FB-VEM) algorithm is developed to tackle the challenging computational problem. Useful theoretical properties are established under certain asymptotic regimes. The proposed method is applied to a complex problem-solving item in the 2012 version of the Programme for International Student Assessment (PISA).
Model-free Estimation of Latent Structure via Multiscale Nonparametric Maximum Likelihood

Bryon Aragam,Ruiyi Yang

2024-10-30

Abstract:Multivariate distributions often carry latent structures that are difficult to identify and estimate, and which better reflect the data generating mechanism than extrinsic structures exhibited simply by the raw data. In this paper, we propose a model-free approach for estimating such latent structures whenever they are present, without assuming they exist a priori. Given an arbitrary density $p_0$, we construct a multiscale representation of the density and propose data-driven methods for selecting representative models that capture meaningful discrete structure. Our approach uses a nonparametric maximum likelihood estimator to estimate the latent structure at different scales and we further characterize their asymptotic limits. By carrying out such a multiscale analysis, we obtain coarseto-fine structures inherent in the original distribution, which are integrated via a model selection procedure to yield an interpretable discrete representation of it. As an application, we design a clustering algorithm based on the proposed procedure and demonstrate its effectiveness in capturing a wide range of latent structures.

Methodology,Statistics Theory,Machine Learning
Mixed Likelihood Gaussian Process Latent Variable Model

Samuel Murray,Hedvig Kjellström

DOI: https://doi.org/10.48550/arXiv.1811.07627

2018-11-19

Abstract:We present the Mixed Likelihood Gaussian process latent variable model (GP-LVM), capable of modeling data with attributes of different types. The standard formulation of GP-LVM assumes that each observation is drawn from a Gaussian distribution, which makes the model unsuited for data with e.g. categorical or nominal attributes. Our model, for which we use a sampling based variational inference, instead assumes a separate likelihood for each observed dimension. This formulation results in more meaningful latent representations, and give better predictive performance for real world data with dimensions of different types.

Machine Learning
Flexible Bayesian Modeling for Longitudinal Binary and Ordinal Responses

Jizhou Kang,Athanasios Kottas

DOI: https://doi.org/10.48550/arXiv.2307.00224

2023-07-01

Methodology

Abstract:Longitudinal studies with binary or ordinal responses are widely encountered in various disciplines, where the primary focus is on the temporal evolution of the probability of each response category. Traditional approaches build from the generalized mixed effects modeling framework. Even amplified with nonparametric priors placed on the fixed or random effects, such models are restrictive due to the implied assumptions on the marginal expectation and covariance structure of the responses. We tackle the problem from a functional data analysis perspective, treating the observations for each subject as realizations from subject-specific stochastic processes at the measured times. We develop the methodology focusing initially on binary responses, for which we assume the stochastic processes have Binomial marginal distributions. Leveraging the logits representation, we model the discrete space processes through sequences of continuous space processes. We utilize a hierarchical framework to model the mean and covariance kernel of the continuous space processes nonparametrically and simultaneously through a Gaussian process prior and an Inverse-Wishart process prior, respectively. The prior structure results in flexible inference for the evolution and correlation of binary responses, while allowing for borrowing of strength across all subjects. The modeling approach can be naturally extended to ordinal responses. Here, the continuation-ratio logits factorization of the multinomial distribution is key for efficient modeling and inference, including a practical way of dealing with unbalanced longitudinal data. The methodology is illustrated with synthetic data examples and an analysis of college students' mental health status data.
Multilevel latent Gaussian process model for mixed discrete and continuous multivariate response data

Erin M. Schliep,Jennifer A. Hoeting

DOI: https://doi.org/10.1007/s13253-013-0136

2013-03-24

Abstract:We propose a Bayesian model for mixed ordinal and continuous multivariate data to evaluate a latent spatial Gaussian process. Our proposed model can be used in many contexts where mixed continuous and discrete multivariate responses are observed in an effort to quantify an unobservable continuous measurement. In our example, the latent, or unobservable measurement is wetland condition. While predicted values of the latent wetland condition variable produced by the model at each location do not hold any intrinsic value, the relative magnitudes of the wetland condition values are of interest. In addition, by including point-referenced covariates in the model, we are able to make predictions at new locations for both the latent random variable and the multivariate response. Lastly, the model produces ranks of the multivariate responses in relation to the unobserved latent random field. This is an important result as it allows us to determine which response variables are most closely correlated with the latent variable. Our approach offers an alternative to traditional indices based on best professional judgment that are frequently used in ecology. We apply our model to assess wetland condition in the North Platte and Rio Grande River Basins in Colorado. The model facilitates a comparison of wetland condition at multiple locations and ranks the importance of in-field measurements.

Methodology
A copula formulation for multivariate latent Markov models

Russo, Alfonso,Farcomeni, Alessio

DOI: https://doi.org/10.1007/s11749-024-00919-9

2024-02-08

Test

Abstract:We specify a general formulation for multivariate latent Markov models for panel data, where outcomes are possibly of mixed-type (categorical, discrete, continuous). Conditionally on a time-varying discrete latent variable and covariates, the joint distribution of outcomes simultaneously observed is expressed through a parametric copula. We therefore do not make any conditional independence assumption. The observed likelihood is maximized by means of an expectation–maximization algorithm. In a simulation study, we argue how modeling the residual contemporary dependence might be crucial in order to avoid bias in the parameter estimates. We illustrate through an original application to assessment of poverty through direct and indirect indicators in a cohort of Italian households.

statistics & probability
Modeling Heterogeneity in Temporal Dynamics: Extending Latent State-Trait Autoregressive and Cross-lagged Panel Models to Mixture Distribution Models

Jana Holtmann,Michael Eid,Philip S Santangelo,Tobias D Kockler,Ulrich W Ebner-Priemer

DOI: https://doi.org/10.1080/00273171.2023.2201824

Abstract:Longitudinal models suited for the analysis of panel data, such as cross-lagged panel or autoregressive latent-state trait models, assume population homogeneity with respect to the temporal dynamics of the variables under investigation. This assumption is likely to be too restrictive in a myriad of research areas. We propose an extension of autoregressive and cross-lagged latent state-trait models to mixture distribution models. The models allow researchers to model unobserved person heterogeneity and qualitative differences in longitudinal dynamics based on comparatively few observations per person, while taking into account temporal dependencies between observations as well as measurement error in the variables. The models are extended to include categorical covariates, to investigate the distribution of encountered latent classes across observed groups. The potential of the models is illustrated with an application to self-esteem and affect data in patients with borderline personality disorder, an anxiety disorder, and healthy control participants. Requirements for the models' applicability are investigated in an extensive simulation study and recommendations for model applications are derived.
A Multiscale Perspective on Maximum Marginal Likelihood Estimation

O. Deniz Akyildiz,Michela Ottobre,Iain Souttar

2024-06-11

Abstract:In this paper, we provide a multiscale perspective on the problem of maximum marginal likelihood estimation. We consider and analyse a diffusion-based maximum marginal likelihood estimation scheme using ideas from multiscale dynamics. Our perspective is based on stochastic averaging; we make an explicit connection between ideas in applied probability and parameter inference in computational statistics. In particular, we consider a general class of coupled Langevin diffusions for joint inference of latent variables and parameters in statistical models, where the latent variables are sampled from a fast Langevin process (which acts as a sampler), and the parameters are updated using a slow Langevin process (which acts as an optimiser). We show that the resulting system of stochastic differential equations (SDEs) can be viewed as a two-time scale system. To demonstrate the utility of such a perspective, we show that the averaged parameter dynamics obtained in the limit of scale separation can be used to estimate the optimal parameter, within the strongly convex setting. We do this by using recent uniform-in-time non-asymptotic averaging bounds. Finally, we conclude by showing that the slow-fast algorithm we consider here, termed Slow-Fast Langevin Algorithm, performs on par with state-of-the-art methods on a variety of examples. We believe that the stochastic averaging approach we provide in this paper enables us to look at these algorithms from a fresh angle, as well as unlocking the path to develop and analyse new methods using well-established averaging principles.

Computation,Probability
Mixture of Latent Trait Analyzers for Model-Based Clustering of Categorical Data

Isabella Gollini,Thomas Brendan Murphy

DOI: https://doi.org/10.48550/arXiv.1301.2167

2013-02-20

Abstract:Model-based clustering methods for continuous data are well established and commonly used in a wide range of applications. However, model-based clustering methods for categorical data are less standard. Latent class analysis is a commonly used method for model-based clustering of binary data and/or categorical data, but due to an assumed local independence structure there may not be a correspondence between the estimated latent classes and groups in the population of interest. The mixture of latent trait analyzers model extends latent class analysis by assuming a model for the categorical response variables that depends on both a categorical latent class and a continuous latent trait variable; the discrete latent class accommodates group structure and the continuous latent trait accommodates dependence within these groups. Fitting the mixture of latent trait analyzers model is potentially difficult because the likelihood function involves an integral that cannot be evaluated analytically. We develop a variational approach for fitting the mixture of latent trait models and this provides an efficient model fitting strategy. The mixture of latent trait analyzers model is demonstrated on the analysis of data from the National Long Term Care Survey (NLTCS) and voting in the U.S. Congress. The model is shown to yield intuitive clustering results and it gives a much better fit than either latent class analysis or latent trait analysis alone.

Methodology,Computation
Anomaly Detection For Categorical Observations Using Latent Gaussian Process

Fengmao Lv,Guowu Yang,Jinzhao Wu,Chuan Liu,Yuhong Yang

DOI: https://doi.org/10.1007/978-3-319-70139-4_29

2017-01-01

Abstract:Anomaly detection is an important problem in many applications, ranging from medical informatics to network security. Various distribution-based techniques have been proposed to tackle this issue, which try to learn the probabilistic distribution of conventional behaviors and consider the observations with low densities as anomalies. For categorical observations, multinomial or dirichlet compound multinomial distributions were adopted as effective statistical models for conventional samples. However, when faced with small-scale data set containing multivariate categorical samples, these models will suffer from the curse of dimensionality and fail to capture the statistical properties of conventional behavior, since only a small proportion of possible categorical configurations will exist in the training data. As an effective bayesian nonparametric technique, categorical latent Gaussian process is able to model small-scale categorical data through learning a continuous latent space for multivariate categorical samples with Gaussian process. Therefore, on the basis of categorical latent Gaussian process, we propose an anomaly detection technique for multivariate categorical observations. In our method, categorical latent Gaussian process is adopted to capture the probabilistic distributions of conventional categorical samples. Experimental results on categorical data set show that our method can effectively detect anomalous categorical observations and achieve better detection performance compared with other anomaly detection techniques.

A Stochastic Multivariate Latent Variable Model For Categorical Responses

Some Remarks on Latent Variable Models in Categorical Data Analysis

A Latent-Variable Bayesian Nonparametric Regression Model

Stochastic Differential Equations with Variational Wishart Diffusions

Modeling Multivariate Mixed-Response Functional Data

Bayesian Analysis of Binary and Polychotomous Response Data

A Fused Latent and Graphical Model for Multivariate Binary Data

Bayesian Semiparametric Longitudinal Inverse-Probit Mixed Models for Category Learning

Multivariate Matérn Models -- A Spectral Approach

Generative Classification Model for Categorical Data Based on Latent Gaussian Process.

A flexible Bayesian generalized linear model for dichotomous response data with an application to text categorization

A Latent Topic Model with Markov Transition for Process Data

Model-free Estimation of Latent Structure via Multiscale Nonparametric Maximum Likelihood

Mixed Likelihood Gaussian Process Latent Variable Model

Flexible Bayesian Modeling for Longitudinal Binary and Ordinal Responses

Multilevel latent Gaussian process model for mixed discrete and continuous multivariate response data

A copula formulation for multivariate latent Markov models

Modeling Heterogeneity in Temporal Dynamics: Extending Latent State-Trait Autoregressive and Cross-lagged Panel Models to Mixture Distribution Models

A Multiscale Perspective on Maximum Marginal Likelihood Estimation

Mixture of Latent Trait Analyzers for Model-Based Clustering of Categorical Data

Anomaly Detection For Categorical Observations Using Latent Gaussian Process