Laplacian Regularized Gaussian Mixture Model for Data Clustering

HeXiaofei,CaiDeng,ShaoYuanlong,BaoHujun,HanJiawei

IF: 9.235

2011-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Gaussian Mixture Models (GMMs) are among the most statistically mature methods for clustering. Each cluster is represented by a Gaussian distribution. The clustering process thereby turns to estima...

What problem does this paper attempt to address?

Laplacian Regularized Gaussian Mixture Model for Data Clustering

Xiaofei He,Deng Cai,Yuanlong Shao,Hujun Bao,Jiawei Han

DOI: https://doi.org/10.1109/tkde.2010.259

IF: 9.235

2011-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Gaussian Mixture Models (GMMs) are among the most statistically mature methods for clustering. Each cluster is represented by a Gaussian distribution. The clustering process thereby turns to estimate the parameters of the Gaussian mixture, usually by the Expectation-Maximization algorithm. In this paper, we consider the case where the probability distribution that generates the data is supported on a submanifold of the ambient space. It is natural to assume that if two points are close in the intrinsic geometry of the probability distribution, then their conditional probability distributions are similar. Specifically, we introduce a regularized probabilistic model based on manifold structure for data clustering, called Laplacian regularized Gaussian Mixture Model (LapGMM). The data manifold is modeled by a nearest neighbor graph, and the graph structure is incorporated in the maximum likelihood objective function. As a result, the obtained conditional probability distribution varies smoothly along the geodesics of the data manifold. Experimental results on real data sets demonstrate the effectiveness of the proposed approach.
Gaussian mixture density modeling and decomposition with weighted likelihood

Xiaobing Yang,FanSheng Kong,Weihua Xu,BiHong Liu

DOI: https://doi.org/10.1109/WCICA.2004.1342311

2004-01-01

Abstract:A classic yet challenging research topic, Gaussian mixture density modeling and decomposition, is attracting much attention in a variety of disciplines. A Gaussian mixture density can be viewed as a contaminated Gaussian density with respect to each Gaussian component in the mixture. A component may not be well represented when the data is analyzed from a mixture density. In this article, Gaussian mixture density modeling and decomposition with weighted likelihood is studied. Then based on the Gaussian mixture density decomposition (GMDD) algorithm and weighted likelihood equations (WLEs), a new algorithm, called weighted Gaussian mixture density decomposition (WGMDD) algorithm is proposed. The WGMDD algorithm is effective and robust. It produces estimates with low bias and mean square error, and makes the applications of GMDD algorithm more extensive and more exact. A small simulation experiment illustrates the above point.
Gaussian Mixture Model with Local Consistency

Jialu Liu,Deng Cai,Xiaofei He

DOI: https://doi.org/10.1609/aaai.v24i1.7659

2010-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Gaussian Mixture Model (GMM) is one of the most popular data clustering methods which can be viewed as a linear combination of different Gaussian components. In GMM, each cluster obeys Gaussian distribution and the task of clustering is to group observations into different components through estimating each cluster's own parameters. The Expectation-Maximization algorithm is always involved in such estimation problem. However, many previous studies have shown naturally occurring data may reside on or close to an underlying submanifold. In this paper, we consider the case where the probability distribution is supported on a submanifold of the ambient space. We take into account the smoothness of the conditional probability distribution along the geodesics of data manifold. That is, if two observations are "close" in intrinsic geometry, their distributions over different Gaussian components are similar. Simply speaking, we introduce a novel method based on manifold structure for data clustering, called Locally Consistent Gaussian Mixture Model (LCGMM). Specifically, we construct a nearest neighbor graph and adopt Kullback-Leibler Divergence as the "distance" measurement to regularize the objective function of GMM. Experiments on several data sets demonstrate the effectiveness of such regularization.
A General Transfer Learning-based Gaussian Mixture Model for Clustering

Rongrong Wang,Jin Zhou,Hui Jiang,Shiyuan Han,Lin Wang,Dong Wang,Yuehui Chen

DOI: https://doi.org/10.1007/s40815-020-01016-3

IF: 4.085

2021-02-05

International Journal of Fuzzy Systems

Abstract:Gaussian mixture model (GMM) is a well-known model-based approach for data clustering. However, when the data samples are insufficient, the classical GMM-based clustering algorithms are not effective anymore. Referring to the idea of transfer clustering methods, this paper proposes a general transfer GMM-based clustering framework, which employs the important knowledge extracted from some known source domain to guide and improve the clustering on the target domain with small-scale data. Specifically, three traditional GMM-based clustering approaches are extended to the corresponding transfer clustering versions. Furthermore, to avoid the negative transfer problem, maximum mean discrepancy (MMD) is introduced to search the most matched source domain to provide more positive guidance for data clustering on the target domain. Experiments on synthetic and real-world datasets demonstrate the efficiency of the presented framework compared with several existing transfer clustering algorithms.

computer science, information systems,automation & control systems, artificial intelligence
Refining Gaussian Mixture Model Based on Enhanced Manifold Learning.

Jianfeng Shen,Jiajun Bu,Bin Ju,Tao Jiang,Hao Wu,Lanjuan Li

DOI: https://doi.org/10.1016/j.neucom.2012.01.029

IF: 6

2012-01-01

Neurocomputing

Abstract:Gaussian mixture model (GMM) has been widely used for data analysis in various domains including text documents, face images and genes. GMM can be viewed as a simple linear superposition of Gaussian components, each of which represents a data cluster. Recent models, namely Laplacian regularized GMM (LapGMM) and locally consistent GMM (LCGMM) have been proposed to preserve the local manifold structure of the data for modeling Gaussian mixtures, and show superior performance than the original GMM. However, these two models ignore the global manifold structure without consideration of the widely separated points. In this paper, we introduce refined Gaussian mixture model (RGMM), which explicitly places separated points far apart from each other as well as brings nearby points closer together according to the probability distributions of Gaussians, in the hope of fully discovering the discriminating power of manifold learning for estimating Gaussian mixtures. We use EM algorithm to optimize the maximum likelihood function of RGMM. Experimental results on three real-world data sets demonstrate the effectiveness of RGMM in data clustering.
Mixed data Deep Gaussian Mixture Model: A clustering model for mixed datasets

Robin Fuchs,Denys Pommeret,Cinzia Viroli

DOI: https://doi.org/10.48550/arXiv.2010.06661

IF: 5.414

2020-10-13

Machine Learning

Abstract:Clustering mixed data presents numerous challenges inherent to the very heterogeneous nature of the variables. A clustering algorithm should be able, despite of this heterogeneity, to extract discriminant pieces of information from the variables in order to design groups. In this work we introduce a multilayer architecture model-based clustering method called Mixed Deep Gaussian Mixture Model (MDGMM) that can be viewed as an automatic way to merge the clustering performed separately on continuous and non-continuous data. This architecture is flexible and can be adapted to mixed as well as to continuous or non-continuous data. In this sense we generalize Generalized Linear Latent Variable Models and Deep Gaussian Mixture Models. We also design a new initialisation strategy and a data driven method that selects the best specification of the model and the optimal number of clusters for a given dataset "on the fly". Besides, our model provides continuous low-dimensional representations of the data which can be a useful tool to visualize mixed datasets. Finally, we validate the performance of our approach comparing its results with state-of-the-art mixed data clustering models over several commonly used datasets.
Mixture of GANs for Clustering.

Yang Yu,Wen-Ji Zhou

DOI: https://doi.org/10.24963/ijcai.2018/423

2018-01-01

Abstract:For data clustering, Gaussian mixture model (GMM) is a typical method that trains several Gaussian models to capture the data. Each Gaussian model then provides the distribution information of a cluster. For clustering of high dimensional and complex data, more flexible models rather than Gaussian models are desired. Recently, the generative adversarial networks (GANs) have shown effectiveness in capturing complex data distribution. Therefore, GAN mixture model (GANMM) would be a promising alternative of GMM. However, we notice that the non-flexibility of the Gaussian model is essential in the expectation-maximization procedure for training GMM. GAN can have much higher flexibility, which disables the commonly employed expectation-maximization procedure, as that the maximization cannot change the result of the expectation. In this paper, we propose to use the epsilon-expectation-maximization procedure for training GANMM. The experiments show that the proposed GANMM can have good performance on complex data as well as simple data.
Gaussian Mixture Model Clustering with Incomplete Data

Yi Zhang,Miaomiao Li,Siwei Wang,Sisi Dai,Lei Luo,En Zhu,Huiying Xu,Xinzhong Zhu,Chaoyun Yao,Haoran Zhou

DOI: https://doi.org/10.1145/3408318

2021-03-31

Abstract:Gaussian mixture model (GMM) clustering has been extensively studied due to its effectiveness and efficiency. Though demonstrating promising performance in various applications, it cannot effectively address the absent features among data, which is not uncommon in practical applications. In this article, different from existing approaches that first impute the absence and then perform GMM clustering tasks on the imputed data, we propose to integrate the imputation and GMM clustering into a unified learning procedure. Specifically, the missing data is filled by the result of GMM clustering, and the imputed data is then taken for GMM clustering. These two steps alternatively negotiate with each other to achieve optimum. By this way, the imputed data can best serve for GMM clustering. A two-step alternative algorithm with proved convergence is carefully designed to solve the resultant optimization problem. Extensive experiments have been conducted on eight UCI benchmark datasets, and the results have validated the effectiveness of the proposed algorithm.
Model-based clustering based on sparse finite Gaussian mixtures

Gertraud Malsiner-Walli,Sylvia Frühwirth-Schnatter,Bettina Grün

DOI: https://doi.org/10.1007/s11222-014-9500-2

2016-06-22

Abstract:In the framework of Bayesian model-based clustering based on a finite mixture of Gaussian distributions, we present a joint approach to estimate the number of mixture components and identify cluster-relevant variables simultaneously as well as to obtain an identified model. Our approach consists in specifying sparse hierarchical priors on the mixture weights and component means. In a deliberately overfitting mixture model the sparse prior on the weights empties superfluous components during MCMC. A straightforward estimator for the true number of components is given by the most frequent number of non-empty components visited during MCMC sampling. Specifying a shrinkage prior, namely the normal gamma prior, on the component means leads to improved parameter estimates as well as identification of cluster-relevant variables. After estimating the mixture model using MCMC methods based on data augmentation and Gibbs sampling, an identified model is obtained by relabeling the MCMC output in the point process representation of the draws. This is performed using $K$-centroids cluster analysis based on the Mahalanobis distance. We evaluate our proposed strategy in a simulation setup with artificial data and by applying it to benchmark data sets.

Methodology
Flexible Clustering with a Sparse Mixture of Generalized Hyperbolic Distributions

Alexa A. Sochaniwsky,Michael P. B. Gallaugher,Yang Tang,Paul D. McNicholas

DOI: https://doi.org/10.1007/s00357-024-09479-x

IF: 1.333

2024-07-13

Journal of Classification

Abstract:Robust clustering of high-dimensional data is an important topic because clusters in real datasets are often heavy-tailed and/or asymmetric. Traditional approaches to model-based clustering often fail for high dimensional data, e.g., due to the number of free covariance parameters. A parametrization of the component scale matrices for the mixture of generalized hyperbolic distributions is proposed. This parameterization includes a penalty term in the likelihood. An analytically feasible expectation-maximization algorithm is developed by placing a gamma-lasso penalty constraining the concentration matrix. The proposed methodology is investigated through simulation studies and illustrated using two real datasets.

mathematics, interdisciplinary applications,psychology, mathematical
Subspace Clustering By Mixture Of Gaussian Regression

Baohua Li,Ying Zhang,Zhouchen Lin,Huchuan Lu

DOI: https://doi.org/10.1109/cvpr.2015.7298821

2015-01-01

Abstract:Subspace clustering is a problem of finding a multi-subspace representation that best fits sample points drawn from a high-dimensional space. The existing clustering models generally adopt different norms to describe noise, which is equivalent to assuming that the data are corrupted by specific types of noise. In practice, however, noise is much more complex. So it is inappropriate to simply use a certain norm to model noise. Therefore, we propose Mixture of Gaussian Regression (MoG Regression) for subspace clustering by modeling noise as a Mixture of Gaussians (MoG). The MoG Regression provides an effective way to model a much broader range of noise distributions. As a result, the obtained affinity matrix is better at characterizing the structure of data in real applications. Experimental results on multiple datasets demonstrate that MoG Regression significantly outperforms state-of-the-art subspace clustering methods.
Gaussian mixture model with an extended ultrametric covariance structure

Carlo Cavicchia,Maurizio Vichi,Giorgia Zaccaria

DOI: https://doi.org/10.1007/s11634-021-00488-x

2022-02-25

Advances in Data Analysis and Classification

Abstract:Gaussian Mixture Models (GMMs) are one of the most widespread methodologies for model-based clustering. They assume a multivariate Gaussian distribution for each component of the mixture, centered at the mean vector and with volume, shape and orientation derived by the covariance matrix. To reduce the large number of parameters produced by the covariance matrices, parsimonious parameterizations of the latter were proposed in literature, e.g., the eigen-decomposition and the parsimonious GMMs based on mixtures of probabilistic principal component analyzers and mixtures of factor analyzers. We introduce a new parameterization of a covariance matrix by defining an extended ultrametric covariance matrix and we implement it into a GMM. This structure can be used to describe multidimensional phenomena which are characterized by nested latent concepts having different levels of abstraction, from the most specific to the most general. The proposal is able to pinpoint a hierarchical structure on variables for each component of the GMM, thus identifying a different characterization of a multidimensional phenomenon for each component (cluster, subpopulation) of the mixture. At the same time, it defines a new parsimonious GMM since the ultrametric covariance structure reconstructs the relationships among variables with a limited number of parameters. The proposal is applied on synthetic and real data. On the former it shows good performance in terms of classification when compared to the other existing parameterizations, and on the latter it also provides insight into the hierarchical relationships among the variables for each cluster.

statistics & probability
Learning Gaussian Mixture Models By Structural Risk Minimization

Liwei Wang,JuFu Feng

DOI: https://doi.org/10.1109/icmlc.2005.1527798

2005-01-01

Abstract:Gaussian mixture models are often used for probability density estimation in pattern recognition and machine learning systems. Selecting an optimal number of components in mixture model is important to ensure an accurate and efficient estimate. In this paper, a methodology based on structural risk minimization is presented which trades off between training error and the model complexity. The main contribution of this work is that we give the capacity of an N-component GMM. When applied to unsupervised learning and speech recognition system, the new method shows good performance compared to classical model selection methods.
Probabilistic Cluster Structure Ensemble

Zhiwen Yu,Le Li,Hau-San Wong,Jane You,Guoqiang Han,Yunjun Gao,Guoxian Yu

DOI: https://doi.org/10.1016/j.ins.2014.01.030

IF: 8.1

2014-01-01

Information Sciences

Abstract:Cluster structure ensemble focuses on integrating multiple cluster structures extracted from different datasets into a unified cluster structure, instead of aligning the individual labels from the clustering solutions derived from multiple homogenous datasets in the cluster ensemble framework. In this article, we design a novel probabilistic cluster structure ensemble framework, referred to as Gaussian mixture model based cluster structure ensemble framework (GMMSE), to identify the most representative cluster structure from the dataset. Specifically, GMMSE first applies the bagging approach to produce a set of variant datasets. Then, a set of Gaussian mixture models are used to capture the underlying cluster structures of the datasets. GMMSE applies K-means to initialize the values of the parameters of the Gaussian mixture model, and adopts the Expectation Maximization approach (EM) to estimate the parameter values of the model. Next, the components of the Gaussian mixture models are viewed as new data samples which are used to construct the representative matrix capturing the relationships among components. The similarity between two components corresponding to their respective Gaussian distributions is measured by the Bhattycharya distance function. Afterwards, GMMSE constructs a graph based on the new data samples and the representative matrix, and searches for the most representative cluster structure. Finally, we also design four criteria to assign the data samples to their corresponding clusters based on the unified cluster structure. The experimental results show that (i) GMMSE works well on synthetic datasets and real datasets in the UCI machine learning repository. (ii) GMMSE outperforms most of the previous cluster ensemble approaches.
Regularization and optimization in model-based clustering

Raphael Araujo Sampaio,Joaquim Dias Garcia,Marcus Poggi,Thibaut Vidal

DOI: https://doi.org/10.1016/j.patcog.2024.110310

IF: 8

2024-02-06

Pattern Recognition

Abstract:Due to their conceptual simplicity, k-means algorithm variants have been extensively used for unsupervised cluster analysis. However, one main shortcoming of these algorithms is that they essentially fit a mixture of identical spherical Gaussians to data that vastly deviates from such a distribution. In comparison, general Gaussian Mixture Models (GMMs) can fit richer structures but require estimating a quadratic number of parameters per cluster to represent the covariance matrices. This poses two main issues: (i) the underlying optimization problems are challenging due to their larger number of local minima, and (ii) their solutions can overfit the data. In this work, we design search strategies that circumvent both issues. We develop more effective optimization algorithms for general GMMs, and we combine these algorithms with regularization strategies that avoid overfitting. Through extensive computational analyses, we observe that optimization or regularization in isolation does not substantially improve cluster recovery. However, combining these techniques permits a completely new level of performance previously unachieved by k-means algorithm variants, unraveling vastly different cluster structures. These results shed new light on the current status quo between GMM and k-means methods and suggest the more frequent use of general GMMs for data exploration. To facilitate such applications, we provide open-source code as well as Julia packages ( UnsupervisedClustering.jl and RegularizedCovarianceMatrices.jl ) implementing the proposed techniques.

computer science, artificial intelligence,engineering, electrical & electronic
Active Curve Axis Gaussian Mixture Models

BB Zhang,CS Zhang,X Yi

DOI: https://doi.org/10.1016/j.patcog.2005.01.017

IF: 8

2005-01-01

Pattern Recognition

Abstract:Gaussian Mixture Models (GMM) have been broadly applied for the fitting of probability density function. However, due to the intrinsic linearity of GMM, usually many components are needed to appropriately fit the data distribution, when there are curve manifolds in the data cloud.
Local Minima Structures in Gaussian Mixture Models

Yudong Chen,Dogyoon Song,Xumei Xi,Yuqian Zhang

DOI: https://doi.org/10.1109/tit.2024.3374716

IF: 2.5

2024-01-01

IEEE Transactions on Information Theory

Abstract:We investigate the landscape of the negative log-likelihood function of Gaussian Mixture Models (GMMs) with a general number of components in the population limit. As the objective function is non-convex, there can exist multiple spurious local minima that are not globally optimal, even for well-separated mixture models. Our study reveals that all local minima share a common structure that partially identifies the cluster centers (i.e., means of the Gaussian components) of the true location mixture. Specifically, each local minimum can be represented as a non-overlapping combination of two types of sub-configurations: (1) fitting a single mean estimate to multiple Gaussian components or (2) fitting multiple estimates to a single true component. These results apply to settings where the true mixture components satisfy a certain separation condition, and are valid even when the number of components is over- or under-specified. We also present a more fine-grained analysis for the setting of one-dimensional GMMs with three components, which provide sharper approximation error bounds with improved dependence on the separation parameter.

computer science, information systems,engineering, electrical & electronic
Unsupervised Classification Based on Penalized Maximum Likelihood of Gaussian Mixture Models

Yu Peng,Feng Jufu

DOI: https://doi.org/10.3969/j.issn.1001-4268.2008.05.003

2008-01-01

Abstract:In this paper we propose an unsupervised classification algorithm which is based on Gaussian mixture models. Thinking that EM algorithm will result in a local optimal resolution of Gaussian mixture models in parameter estimations, we substitute invert Wishart distribution for Jeffery prior. Experiments show that this algorithm improves correct rates and decreases time while estimating classifications.
The parsimonious Gaussian mixture models with partitioned parameters and their application in clustering

Niloofar Aslani Akhore Olyaei,Mojtaba Khazaei,Dariush Najarzadeh

DOI: https://doi.org/10.1007/s10260-023-00743-9

2024-01-26

Statistical Methods & Applications

Abstract:Cluster analysis is a method that identifies similar groups of data without any prior knowledge of the relevant groups. One of the most widely used clustering methods is model-based clustering, in which data clustering is performed by fitting a probabilistic model to the data. Mixture of Gaussian distributions is a commonly used model in model-based clustering. Unfortunately, the number of covariance matrices parameters rapidly increases by increasing the number of variables or components in these models. So far, various classes of the parsimonious Gaussian mixture models, by applying various constraints on the covariance matrices, have been introduced to solve this problem. Unfortunately, the number of models in each of these classes is so small such that in practice it does not allow the study and selection of models with any number of parameters, which can vary between the minimum number (one parameter) and the maximum number (no constraints model) of parameters. In this paper, to deal with this problem a family of the parsimonious Gaussian mixture models is introduced. This is done by identifying and determining the appropriate partitions of the variances and correlation coefficients between variables among clusters. We call these models "the parsimonious Gaussian mixture models with partitioned parameters". The generalized Expectation-Conditional Maximization algorithm, by employing the Fisher scoring method within the algorithm, is used to compute the maximum likelihood estimates of parameters. Bayesian information criterion is used for comparing and selecting the best model. Also, the steepest ascent method is adapted to search the best model. Finally, performances of these models are examined on two real datasets and a brief simulation study.

statistics & probability

Laplacian Regularized Gaussian Mixture Model for Data Clustering

Laplacian Regularized Gaussian Mixture Model for Data Clustering

Gaussian mixture density modeling and decomposition with weighted likelihood

Gaussian Mixture Model with Local Consistency

A General Transfer Learning-based Gaussian Mixture Model for Clustering

Refining Gaussian Mixture Model Based on Enhanced Manifold Learning.

Mixed data Deep Gaussian Mixture Model: A clustering model for mixed datasets

Mixture of GANs for Clustering.

Gaussian Mixture Model Clustering with Incomplete Data

Model-based clustering based on sparse finite Gaussian mixtures

Flexible Clustering with a Sparse Mixture of Generalized Hyperbolic Distributions

Subspace Clustering By Mixture Of Gaussian Regression

Gaussian mixture model with an extended ultrametric covariance structure

Learning Gaussian Mixture Models By Structural Risk Minimization

Probabilistic Cluster Structure Ensemble

Regularization and optimization in model-based clustering

Active Curve Axis Gaussian Mixture Models

Local Minima Structures in Gaussian Mixture Models

Unsupervised Classification Based on Penalized Maximum Likelihood of Gaussian Mixture Models

The parsimonious Gaussian mixture models with partitioned parameters and their application in clustering