Active Curve Axis Gaussian Mixture Models

BB Zhang,CS Zhang,X Yi

DOI: https://doi.org/10.1016/j.patcog.2005.01.017

IF: 8

2005-01-01

Pattern Recognition

Abstract:Gaussian Mixture Models (GMM) have been broadly applied for the fitting of probability density function. However, due to the intrinsic linearity of GMM, usually many components are needed to appropriately fit the data distribution, when there are curve manifolds in the data cloud.

What problem does this paper attempt to address?

Gaussian mixture density modeling and decomposition with weighted likelihood

Xiaobing Yang,FanSheng Kong,Weihua Xu,BiHong Liu

DOI: https://doi.org/10.1109/WCICA.2004.1342311

2004-01-01

Abstract:A classic yet challenging research topic, Gaussian mixture density modeling and decomposition, is attracting much attention in a variety of disciplines. A Gaussian mixture density can be viewed as a contaminated Gaussian density with respect to each Gaussian component in the mixture. A component may not be well represented when the data is analyzed from a mixture density. In this article, Gaussian mixture density modeling and decomposition with weighted likelihood is studied. Then based on the Gaussian mixture density decomposition (GMDD) algorithm and weighted likelihood equations (WLEs), a new algorithm, called weighted Gaussian mixture density decomposition (WGMDD) algorithm is proposed. The WGMDD algorithm is effective and robust. It produces estimates with low bias and mean square error, and makes the applications of GMDD algorithm more extensive and more exact. A small simulation experiment illustrates the above point.
Gaussian model for closed curves

Krzysztof Byrski,Przemysław Spurek,Jacek Tabor

2023-04-04

Abstract:Gaussian Mixture Models (GMM) do not adapt well to curved and strongly nonlinear data. However, we can use Gaussians in the curvilinear coordinate systems to solve this problem. Moreover, such a solution allows for the adaptation of clusters to the complicated shapes defined by the family of functions. But still, it is challenging to model clusters as closed curves (e.g., circles, ellipses, etc.). In this work, we propose a density representation of the closed curve, which can be used to detect the complicated templates in the data. For this purpose, we define a new probability distribution to model closed curves. Then we construct a mixture of such distributions and show that it can be effectively trained in the case of the one-dimensional closed curves.

Computer Vision and Pattern Recognition
Facial feature points detecting based on Gaussian Mixture Models

Junnan Wang,Rong Xiong,Jian Chu

DOI: https://doi.org/10.1016/j.patrec.2014.11.004

IF: 4.757

2015-01-01

Pattern Recognition Letters

Abstract:We use Gaussian Mixture Models to approximate the confidence maps analytically.We also use Gaussian Mixture Models to model the distribution of face shape.The posterior is maximized by iteratively maximizing a lower bound of it.High-order geometry constraint within facial features are considered.Model based Hough Voting scheme is proposed to estimate an initial face shape. Detecting predefined facial feature points in a human face image is a well studied problem. Despite the impressive achievements that have been made, it is still open under unconstrained environments, with variations of illumination, expression, head pose, as well as partial occlusions. This paper proposed a novel method to locate facial feature points under the variations mentioned above. Support vector machines with probability outputs are trained to provide the observation probability of each facial feature point. The observation probabilities, as well as the distribution of face shape which serves as the prior to constrain the relative position of the facial feature points, are both approximated with Gaussian Mixture Models. The problem is solved by maximizing the posterior which combines the prior and observation probability within the framework of Bayesian Inference. An optimization algorithm is developed to maximize the posterior by iteratively maximizing the lower bound of it. The proposed method preserves the high-order geometric constraint within facial feature points. With a simple initialization method of Model based Hough Voting, the method shows competitive detecting rate and locating accuracy on the LFPW and LFW datasets, compared to the methods of state-of-the-art.
Active Function Cross-Entropy Clustering

P. Spurek,J. Tabor,P. Markowicz

DOI: https://doi.org/10.48550/arXiv.1502.01943

2015-02-07

Abstract:Gaussian Mixture Models (GMM) have found many applications in density estimation and data clustering. However, the model does not adapt well to curved and strongly nonlinear data. Recently there appeared an improvement called AcaGMM (Active curve axis Gaussian Mixture Model), which fits Gaussians along curves using an EM-like (Expectation Maximization) approach. Using the ideas standing behind AcaGMM, we build an alternative active function model of clustering, which has some advantages over AcaGMM. In particular it is naturally defined in arbitrary dimensions and enables an easy adaptation to clustering of complicated datasets along the predefined family of functions. Moreover, it does not need external methods to determine the number of clusters as it automatically reduces the number of groups on-line.

Machine Learning
Laplacian Regularized Gaussian Mixture Model for Data Clustering

Xiaofei He,Deng Cai,Yuanlong Shao,Hujun Bao,Jiawei Han

DOI: https://doi.org/10.1109/tkde.2010.259

IF: 9.235

2011-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Gaussian Mixture Models (GMMs) are among the most statistically mature methods for clustering. Each cluster is represented by a Gaussian distribution. The clustering process thereby turns to estimate the parameters of the Gaussian mixture, usually by the Expectation-Maximization algorithm. In this paper, we consider the case where the probability distribution that generates the data is supported on a submanifold of the ambient space. It is natural to assume that if two points are close in the intrinsic geometry of the probability distribution, then their conditional probability distributions are similar. Specifically, we introduce a regularized probabilistic model based on manifold structure for data clustering, called Laplacian regularized Gaussian Mixture Model (LapGMM). The data manifold is modeled by a nearest neighbor graph, and the graph structure is incorporated in the maximum likelihood objective function. As a result, the obtained conditional probability distribution varies smoothly along the geodesics of the data manifold. Experimental results on real data sets demonstrate the effectiveness of the proposed approach.
Gaussian Mixture Model with Local Consistency

Jialu Liu,Deng Cai,Xiaofei He

DOI: https://doi.org/10.1609/aaai.v24i1.7659

2010-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Gaussian Mixture Model (GMM) is one of the most popular data clustering methods which can be viewed as a linear combination of different Gaussian components. In GMM, each cluster obeys Gaussian distribution and the task of clustering is to group observations into different components through estimating each cluster's own parameters. The Expectation-Maximization algorithm is always involved in such estimation problem. However, many previous studies have shown naturally occurring data may reside on or close to an underlying submanifold. In this paper, we consider the case where the probability distribution is supported on a submanifold of the ambient space. We take into account the smoothness of the conditional probability distribution along the geodesics of data manifold. That is, if two observations are "close" in intrinsic geometry, their distributions over different Gaussian components are similar. Simply speaking, we introduce a novel method based on manifold structure for data clustering, called Locally Consistent Gaussian Mixture Model (LCGMM). Specifically, we construct a nearest neighbor graph and adopt Kullback-Leibler Divergence as the "distance" measurement to regularize the objective function of GMM. Experiments on several data sets demonstrate the effectiveness of such regularization.
Local Minima Structures in Gaussian Mixture Models

Yudong Chen,Dogyoon Song,Xumei Xi,Yuqian Zhang

DOI: https://doi.org/10.1109/TIT.2024.3374716

2024-03-09

Abstract:We investigate the landscape of the negative log-likelihood function of Gaussian Mixture Models (GMMs) with a general number of components in the population limit. As the objective function is non-convex, there can be multiple local minima that are not globally optimal, even for well-separated mixture models. Our study reveals that all local minima share a common structure that partially identifies the cluster centers (i.e., means of the Gaussian components) of the true location mixture. Specifically, each local minimum can be represented as a non-overlapping combination of two types of sub-configurations: fitting a single mean estimate to multiple Gaussian components or fitting multiple estimates to a single true component. These results apply to settings where the true mixture components satisfy a certain separation condition, and are valid even when the number of components is over- or under-specified. We also present a more fine-grained analysis for the setting of one-dimensional GMMs with three components, which provide sharper approximation error bounds with improved dependence on the separation.

Machine Learning,Statistics Theory
A Two-Layer Mixture Model of Gaussian Process Functional Regressions and Its MCMC EM Algorithm

Di Wu,Jinwen Ma

DOI: https://doi.org/10.1109/tnnls.2017.2782711

IF: 14.255

2018-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:The mixture of Gaussian processes (GPs) is capable of learning any general stochastic process based on a given set of (sample) curves for the regression and prediction problems. However, it is ineffective for curve clustering and prediction, when the sample curves are derived from different stochastic processes as independent sources linearly mixed together. In this paper, we propose a two-layer mixture model of GP functional regressions (GPFRs) to describe such a mixture of general stochastic processes or independent sources, especially for curve clustering and prediction. Specifically, in the lower layer, the mixture of GPFRs (MGPFRs) is developed for a cluster (or class) of curves within the input space. In the higher layer, the mixture of MGPFRs is further established to divide the curves into clusters according to its components in the output space. For the parameter estimation of the two-layer mixture of GPFRs, we develop a Monte Carlo EM algorithm based on a Monte Carlo Markov chain (MCMC) method, in short, the MCMC EM algorithm. We validate the hierarchical mixture of GPFRs and MCMC EM algorithm using synthetic and real-world data sets. Our results show that our new model outperforms the conventional mixture models in curve clustering and prediction.
On the properties of Gaussian Copula Mixture Models

Ke Wan,Alain Kornhauser

2023-05-24

Abstract:This paper investigates Gaussian copula mixture models (GCMM), which are an extension of Gaussian mixture models (GMM) that incorporate copula concepts. The paper presents the mathematical definition of GCMM and explores the properties of its likelihood function. Additionally, the paper proposes extended Expectation Maximum algorithms to estimate parameters for the mixture of copulas. The marginal distributions corresponding to each component are estimated separately using nonparametric statistical methods. In the experiment, GCMM demonstrates improved goodness-of-fitting compared to GMM when using the same number of clusters. Furthermore, GCMM has the ability to leverage un-synchronized data across dimensions for more comprehensive data analysis.

Machine Learning
Robust Gaussian Mixture Modeling: A -Divergence Based Approach

Ori Kenig,Koby Todros,Tülay Adali

DOI: https://doi.org/10.1109/tsp.2024.3426965

IF: 4.875

2024-08-21

IEEE Transactions on Signal Processing

Abstract:This paper addresses the problem of robust Gaussian mixture modeling in the presence of outliers. We commence by introducing a general expectation-maximization (EM)-like scheme, called -BM, for iterative numerical computation of the minimum -divergence estimator (M DE). This estimator leverages Parzen's non-parametric ernel density estimate to down-weight low density regions associated with outlying measurements. Akin to the conventional EM, the -BM involves successive Maximizations of lower Bounds on the objective function of the M DE. However, differently from EM, these bounds are not exclusively reliant on conditional expectations. The -BM algorithm is applied to robust parameter estimation of a finite-order multivariate Gaussian mixture model (GMM). We proceed by introducing a new robust variant of the Bayesian information criterion (BIC) that penalizes the M DE's objective function. The proposed criterion, called -BIC, is conveniently applied for robust GMM order selection. In the paper, we also establish a data-driven procedure for selection of the kernel's bandwidth parameter. This procedure operates by minimizing an empirical asymptotic approximation of the mean-integrated-squared-error (MISE) between the underlying density and the estimated GMM density. Lastly, the -BM, the -BIC, and the MISE based selection of the kernel's bandwidth are combined into a unified framework for joint order selection and parameter estimation of a GMM. The advantages of the -divergence based framework over other robust approaches are illustrated in simulation studies involving synthetic and real data.

engineering, electrical & electronic
Mixture of Gaussian Processes and its Applications

Mian Huang,Runze Li,Hansheng Wang,Weixin Yao

2010-01-01

Abstract:The rapid development of information technologies enables researchers to collect and store functional data at a low cost. As a result, the quantitative analysis of functional data becomes practically feasible, which naturally calls for new statistical methods to serve such a purpose. To this end, we propose a new model, namely, “Mixture of Gaussian Processes” in this paper. Our method can be viewed as a natural extension of high-dimensional normal mixtures. However, the key difference is that smoothed structures are imposed for both the mean and covariance functions. As a consequence, our model can be estimated efficiently by a novel combination of the ideas from EM algorithm, kernel regression, and functional principal component analysis. It is remarkable that no high-dimensional covariance matrix needs to be estimated in our computational process. This has been an inevitable step for a general normal mixture model and suffers from its estimation instability and inaccuracy. The proposed methodology is empirically justified by Monte Carlo simulations and illustrated by an analysis of a supermarket dataset is illustrated.
Estimating Mixture of Gaussian Processes by Kernel Smoothing

Mian Huang,Runze Li,Hansheng Wang,Weixin Yao

DOI: https://doi.org/10.2139/ssrn.2356155

2014-01-01

Abstract:When the functional data are not homogeneous, e.g., there exist multiple classes of functional curves in the dataset, traditional estimation methods may fail. In this paper, we propose a new estimation procedure for the Mixture of Gaussian Processes, to incorporate both functional and inhomogeneous properties of the data. Our method can be viewed as a natural extension of high-dimensional normal mixtures. However, the key difference is that smoothed structures are imposed for both the mean and covariance functions. The model is shown to be identifiable, and can be estimated efficiently by a combination of the ideas from EM algorithm, kernel regression, and functional principal component analysis. Our methodology is empirically justified by Monte Carlo simulations and illustrated by an analysis of a supermarket dataset.
Self -Paced Mixture of T Distribution Model.

Yang Zhang,Qingtao Tang,Li Niu,Tao Dai,Xi Xiao,Shu-Tao Xia

DOI: https://doi.org/10.1109/icassp.2018.8462323

2018-01-01

Abstract:Gaussian mixture model (GMM) is a powerful probabilistic model for representing the probability distribution of observations in the population. However, the fitness of Gaussian mixture model can be significantly degraded when the data contain a certain amount of outliers. Although there are certain variants of GMM (e.g., mixture of Laplace, mixture of t distribution) attempting to handle outliers, none of them can sufficiently mitigate the effect of outliers if the outliers are far from the centroids. Aiming to remove the effect of outliers further, this paper introduces a Self-Paced Learning mechanism into mixture of t distribution, which leads to Self-Paced Mixture of t distribution model (SPTMM). We derive an Expectation-Maximization based algorithm to train SPTMM and show SPTMM is able to screen the outliers. To demonstrate the effectiveness of SPTMM, we apply the model to density estimation and clustering. Finally, the results indicate that SPTMM outperforms other methods, especially on the data with outliers.
Mixture Gaussian Process Model with Gaussian Mixture Distribution for Big Data

Yaonan Guan,Shaoying He,Shuangshuang Ren,Shuren Liu,Dewei Li

DOI: https://doi.org/10.1016/j.chemolab.2024.105201

IF: 4.175

2024-01-01

Chemometrics and Intelligent Laboratory Systems

Abstract:In the era of chemical big data, the high complexity and strong interdependencies present in the datasets pose considerable challenges when constructing accurate parametric models. The Gaussian process model, owing to its non-parametric nature, demonstrates better adaptability when confronted with complex and interdependent data. However, the standard Gaussian process has two significant limitations. Firstly, the time complexity of inverting its kernel matrix during the inference process is O(n)3. Secondly, all data share a common kernel function parameter, which mixes different data types and reduces the model accuracy in mixing-category data identification problems. In light of this, this paper proposes a mixture Gaussian process model that addresses these limitations. This model reduces time complexity and distinguishes data based on different data features. It incorporates a Gaussian mixture distribution for the inducing variables to approximate the original data distribution. Stochastic Variational Inference is utilized to reduce the computational time required for parameter inference. The inducing variables have distinct parameters for the kernel function based on the data category, leading to improved analytical accuracy and reduced time complexity of the Gaussian process model. Numerical experiments are conducted to analyze and compare the performance of the proposed model on different-sized datasets and various data category cases.
Learning GMM Using Elliptically Contoured Distributions

Bo Li,Wenju Liu,Lihua Dou

DOI: https://doi.org/10.1109/ICPR.2010.130

IF: 8

2010-01-01

Pattern Recognition

Abstract:Model order selection and parameter estimation for Gaussian mixture model (GMM) are important issues for clustering analysis and density estimation. Most methods for model selection usually add a penalty term in the objective function that can penalize the models and choose an optimal one from a set of candidate models. This paper presents a simple and novel approach to determine the number of components and simultaneously estimate the parameters for GMM. By introducing the degenerating model, the proposed approach overcomes the drawback of likelihood estimate that is a non-decreasing function and can not be used to select the number of components. The degenerating model is a more general form of mixture component density and it can degenerate into the component density or a crater-like density when its parameter K varies from 1 to a bigger value. The likelihood of the crater-like density evaluated for the training data approximates to zero. This characteristic of the degenerating model forms the foundation of the proposed approach. The experimental results show robust and evident performance improvement of the approach.
Gaussian mixture model with an extended ultrametric covariance structure

Carlo Cavicchia,Maurizio Vichi,Giorgia Zaccaria

DOI: https://doi.org/10.1007/s11634-021-00488-x

2022-02-25

Advances in Data Analysis and Classification

Abstract:Gaussian Mixture Models (GMMs) are one of the most widespread methodologies for model-based clustering. They assume a multivariate Gaussian distribution for each component of the mixture, centered at the mean vector and with volume, shape and orientation derived by the covariance matrix. To reduce the large number of parameters produced by the covariance matrices, parsimonious parameterizations of the latter were proposed in literature, e.g., the eigen-decomposition and the parsimonious GMMs based on mixtures of probabilistic principal component analyzers and mixtures of factor analyzers. We introduce a new parameterization of a covariance matrix by defining an extended ultrametric covariance matrix and we implement it into a GMM. This structure can be used to describe multidimensional phenomena which are characterized by nested latent concepts having different levels of abstraction, from the most specific to the most general. The proposal is able to pinpoint a hierarchical structure on variables for each component of the GMM, thus identifying a different characterization of a multidimensional phenomenon for each component (cluster, subpopulation) of the mixture. At the same time, it defines a new parsimonious GMM since the ultrametric covariance structure reconstructs the relationships among variables with a limited number of parameters. The proposal is applied on synthetic and real data. On the former it shows good performance in terms of classification when compared to the other existing parameterizations, and on the latter it also provides insight into the hierarchical relationships among the variables for each cluster.

statistics & probability
Gaussian implementation of the multi-Bernoulli mixture filter

Ángel F. García-Fernández,Yuxuan Xia,Karl Granström,Lennart Svensson,Jason L. Williams

DOI: https://doi.org/10.48550/arXiv.1908.08819

2019-08-23

Abstract:This paper presents the Gaussian implementation of the multi-Bernoulli mixture (MBM) filter. The MBM filter provides the filtering (multi-target) density for the standard dynamic and radar measurement models when the birth model is multi-Bernoulli or multi-Bernoulli mixture. Under linear/Gaussian models, the single target densities of the MBM mixture admit Gaussian closed-form expressions. Murty's algorithm is used to select the global hypotheses with highest weights. The MBM filter is compared with other algorithms in the literature via numerical simulations.

Signal Processing,Computer Vision and Pattern Recognition,Applications

Active Curve Axis Gaussian Mixture Models

Gaussian mixture density modeling and decomposition with weighted likelihood

Gaussian model for closed curves

Facial feature points detecting based on Gaussian Mixture Models

Active Function Cross-Entropy Clustering

Laplacian Regularized Gaussian Mixture Model for Data Clustering

Gaussian Mixture Model with Local Consistency

Local Minima Structures in Gaussian Mixture Models

A Two-Layer Mixture Model of Gaussian Process Functional Regressions and Its MCMC EM Algorithm

On the properties of Gaussian Copula Mixture Models

Robust Gaussian Mixture Modeling: A -Divergence Based Approach

Mixture of Gaussian Processes and its Applications

Estimating Mixture of Gaussian Processes by Kernel Smoothing

Self -Paced Mixture of T Distribution Model.

Mixture Gaussian Process Model with Gaussian Mixture Distribution for Big Data

Learning GMM Using Elliptically Contoured Distributions

Gaussian mixture model with an extended ultrametric covariance structure

Gaussian implementation of the multi-Bernoulli mixture filter