An Effective Em Algorithm for Pca Mixture Model

Z Jin,F Davoine,Z Lou

DOI: https://doi.org/10.1007/978-3-540-27868-9_68

2004-01-01

Abstract:This paper studied PCA mixture model in high dimensional space. A novel EM learning approach by using perturbation was proposed for the PCA mixture model. Experiments showed the novel perturbation EM algorithm is more effective in learning PCA mixture model than an existing constrained EM algorithm.

What problem does this paper attempt to address?

Multimode Process Monitoring with PCA Mixture Model

Xianzhen Xu,Lei Xie,Shuqing Wang

DOI: https://doi.org/10.1016/j.compeleceng.2014.08.002

IF: 4.152

2014-01-01

Computers & Electrical Engineering

Abstract:For multimode processes, Gaussian mixture model (GMM) has been applied to estimate the probability density function of the process data under normal-operational condition in last few years. However, learning GMM with the expectation maximization (EM) algorithm from process data can be difficult or even infeasible for high-dimensional and collinear process variables. To address this issue, a novel multimode process monitoring approach based on PCA mixture model is proposed. First, the PCA technique is directly applied to the covariance matrix of each Gaussian component to reduce the dimension of process variables and to obtain nonsingular covariance matrices. Then the Bayesian Ying-Yang incremental EM algorithm is adopted to automatically optimize the number of mixture components. With the obtained PCA mixture model, a novel process monitoring scheme is derived for fault detection of multimode processes. Three case studies are provided to evaluate the monitoring performance of the proposed method.
Repulsive Mixture Models of Exponential Family PCA for Clustering

Qiao Maoying,Liu Tongliang,Yu Jun,Bian Wei,Tao Dacheng

2020-01-01

Abstract: The mixture extension of exponential family principal component analysis (EPCA) was designed to encode much more structural information about data distribution than the traditional EPCA does. For example, due to the linearity of EPCA's essential form, nonlinear cluster structures cannot be easily handled, but they are explicitly modeled by the mixing extensions. However, the traditional mixture of local EPCAs has the problem of model redundancy, i.e., overlaps among mixing components, which may cause ambiguity for data clustering. To alleviate this problem, in this paper, a repulsiveness-encouraging prior is introduced among mixing components and a diversified EPCA mixture (DEPCAM) model is developed in the Bayesian framework. Specifically, a determinantal point process (DPP) is exploited as a diversity-encouraging prior distribution over the joint local EPCAs. As required, a matrix-valued measure for L-ensemble kernel is designed, within which, $\ell_1$ constraints are imposed to facilitate selecting effective PCs of local EPCAs, and angular based similarity measure are proposed. An efficient variational EM algorithm is derived to perform parameter learning and hidden variable inference. Experimental results on both synthetic and real-world datasets confirm the effectiveness of the proposed method in terms of model parsimony and generalization ability on unseen test data.
Competitive EM Algorithm for Finite Mixture Models

BB Zhang,CS Zhang,X Yi

DOI: https://doi.org/10.1016/s0031-3203(03)00140-7

IF: 8

2004-01-01

Pattern Recognition

Abstract:In this paper, we present a novel competitive EM (CEM) algorithm for finite mixture models to overcome the two main drawbacks of the EM algorithm: often getting trapped at local maxima and sometimes converging to the boundary of the parameter space. The proposed algorithm is capable of automatically choosing the clustering number and selecting the “split” or “merge” operations efficiently based on the new competitive mechanism we propose. It is insensitive to the initial configuration of the mixture component number and model parameters.
An Mcmc Based Em Algorithm For Mixtures Of Gaussian Processes

Di Wu,Ziyi Chen,Jinwen Ma

DOI: https://doi.org/10.1007/978-3-319-25393-0_36

2015-01-01

Abstract:The mixture of Gaussian processes (MGP) is a powerful statistical learning model for regression and prediction and the EM algorithm is an effective method for its parameter learning or estimation. However, the feasible EM algorithms for MGPs are certain approximations of the real EM algorithm since Q-function cannot be computed efficiently in this situation. To overcome this problem, we propose an MCMC based EM algorithm for MGPs where Q-function is alternatively estimated on a set of simulated samples via the Markov Chain Monte Carlo (MCMC) method. It is demonstrated by the experiments on both the synthetic and real-world datasets that our proposed MCMC based EM algorithm is more effective than the other three EM algorithms for MGPs.
On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Babak Barazandeh,Meisam Razaviyayn

DOI: https://doi.org/10.48550/arXiv.1809.08705

2018-09-24

Abstract:Finite mixture models are among the most popular statistical models used in different data science disciplines. Despite their broad applicability, inference under these models typically leads to computationally challenging non-convex problems. While the Expectation-Maximization (EM) algorithm is the most popular approach for solving these non-convex problems, the behavior of this algorithm is not well understood. In this work, we focus on the case of mixture of Laplacian (or Gaussian) distribution. We start by analyzing a simple equally weighted mixture of two single dimensional Laplacian distributions and show that every local optimum of the population maximum likelihood estimation problem is globally optimal. Then, we prove that the EM algorithm converges to the ground truth parameters almost surely with random initialization. Our result extends the existing results for Gaussian distribution to Laplacian distribution. Then we numerically study the behavior of mixture models with more than two components. Motivated by our extensive numerical experiments, we propose a novel stochastic method for estimating the mean of components of a mixture model. Our numerical experiments show that our algorithm outperforms the Naive EM algorithm in almost all scenarios.

Machine Learning
Robust modeling of mixture probabilistic principal component analysis and process monitoring application

jinlin zhu,zhiqiang ge,zhihuan song

DOI: https://doi.org/10.1002/aic.14419

IF: 4.167

2014-01-01

AIChE Journal

Abstract:In this article, a robust modeling strategy for mixture probabilistic principal component analysis (PPCA) is proposed. Different from the traditional Gaussian distribution driven model such as PPCA, the multivariate student t-distribution is adopted for probabilistic modeling to reduce the negative effect of outliers, which is very common in the process industry. Furthermore, for handling the missing data problem, a partially updating algorithm is developed for parameter learning in the robust mixture PPCA model. Therefore, the new robust model can simultaneously deal with outliers and missing data. For process monitoring, a Bayesian soft decision fusion strategy is developed which is combined with the robust local monitoring models under different operating conditions. Two case studies demonstrate that the new robust model shows enhanced modeling and monitoring performance in both outlier and missing data cases, compared to the mixture probabilistic principal analysis model. (c) 2014 American Institute of Chemical Engineers AIChE J, 60: 2143-2157, 2014
HeMPPCAT: Mixtures of Probabilistic Principal Component Analysers for Data with Heteroscedastic Noise

Alec S. Xu,Laura Balzano,Jeffrey A. Fessler

DOI: https://doi.org/10.48550/arXiv.2301.08852

2023-01-26

Abstract:Mixtures of probabilistic principal component analysis (MPPCA) is a well-known mixture model extension of principal component analysis (PCA). Similar to PCA, MPPCA assumes the data samples in each mixture contain homoscedastic noise. However, datasets with heterogeneous noise across samples are becoming increasingly common, as larger datasets are generated by collecting samples from several sources with varying noise profiles. The performance of MPPCA is suboptimal for data with heteroscedastic noise across samples. This paper proposes a heteroscedastic mixtures of probabilistic PCA technique (HeMPPCAT) that uses a generalized expectation-maximization (GEM) algorithm to jointly estimate the unknown underlying factors, means, and noise variances under a heteroscedastic noise setting. Simulation results illustrate the improved factor estimates and clustering accuracies of HeMPPCAT compared to MPPCA.

Methodology,Signal Processing,Machine Learning
An Effective EM Algorithm for Mixtures of Gaussian Processes Via the MCMC Sampling and Approximation.

Di Wu,Jinwen Ma

DOI: https://doi.org/10.1016/j.neucom.2018.11.046

IF: 6

2018-01-01

Neurocomputing

Abstract:The Mixture of Gaussian Processes (MGP) is a powerful statistical model for characterizing multimodal data, but its conventional Expectation-Maximization (EM) algorithm (Dempster et al., 1977) is computationally intractable because of its time complexity. To solve this problem, some approximation techniques have been proposed in the conventional EM algorithm. However, these approximate EM algorithms are ineffective or limited in some situations. To implement the EM algorithm more effectively, we approximate the EM algorithm with simulated samples of latent variable via the Monte Carlo Markov Chain (MCMC) sampling, and design an MCMC EM algorithm. Experiments on both synthetic and real-world data sets demonstrate that our MCMC EM algorithm is more effective than the state-of-the-art EM algorithms on classification and prediction problems. (C) 2018 Elsevier B.V. All rights reserved.
Automatic T-Mixture Model Selection Via Rival Penalized EM

Chunyan Zhang,Jin Tang,Bin Luo

DOI: https://doi.org/10.1109/his.2006.264904

2006-01-01

Abstract:Modelling mixtures of multivariate t-distributions are usually used instead of Gaussian mixture models(GMM) as a robust approach, when one fits a set of continuous multivariate data which have wider tail than Gaussian's or atypical observations, but it is unable to perform model selection automatically through the traditional EM (Expectation Maximization) algorithm. To solve this problem, a new algorithm, which is called Rival Penalized Expectation-Maximization (RPEM) algorithm, is proposed to t-mixture model (TMM). It can automatically select an appropriate number of densities in t-density mixture model. Experimental results on unsupervised color image segmentation demonstrate the affectivity of the proposed algorithm.
An EM Algorithm for Robust Bayesian PCA with Student’s T-Distribution

Jiading Gai,Yong Li,Robert L. Stevenson

DOI: https://doi.org/10.1109/icip.2008.4712344

2008-01-01

Abstract:Principal component analysis (PCA) is a technique that is widely used for applications such as dimensionality reduction, image compression, feature extraction and data visualization. One of the key issues in the use of PCA for modelling is that it is very sensitive to outliers since its formulation is based on Gaussian density model. Lately, more heavy-tailed distribution (i.e., Student’s t-distribution) is introduced to increase the robustness of traditional PCA. But the robust version of PCA is expressed as the maximum likelihood solution of a probabilistic latent variable model. This reformulation raises the question of how to determine the optimal number of principal components to be retained. In this paper, we develop a Bayesian model selection approach to estimate the true dimensionality of the data. The proposed algorithm is based on a new Bayesian treatment of robust Student’s t-distribution PCA. A simple Expectation-Maximization (EM) solver is introduced to find approximate solutions for the model. Experiments show that the proposed model achieves simultaneous optimal dimensionality selection and accurate principal components recovery.
On Semi-Supervised Learning Genetic-Based And Deterministic Annealing Em Algorithm For Dirichlet Mixture Models

jing hua bai,kan li,x x zhang

DOI: https://doi.org/10.4028/www.scientific.net/amm.39.151

2011-01-01

Abstract:We propose a genetic-based and deterministic annealing expectation-maximization (GA&DA-EM) algorithm for learning Dirichlet mixture models from multivariate data. This algorithm is capable of selecting the number of components of the model using the minimum description length (MDL) criterion. Our approach benefits from the properties of Genetic algorithms and deterministic annealing algorithm by combination of both into a single procedure. The population-based stochastic search of the GA&DA explores the search space more thoroughly than the EM method. Therefore, our algorithm enables escaping from local optimal solutions since the algorithm becomes less sensitive to its initialization. The GA&DA-EM algorithm is elitist which maintains the monotonic convergence property of the EM algorithm. We conducted experiments on the WebKB and 20NEWSGROUPS. The results show that show that 1) the GA&DA-EM outperforms the EM method since: Our approach identifies the number of components which were used to generate the underlying data more often than the EM algorithm. 2) the algorithm alternatives to EM that overcoming the challenges of local maxima.
The Competitive EM Algorithm for Gaussian Mixtures with BYY Harmony Criterion

Hengyu Wang,Lei Li,Jinwen Ma

DOI: https://doi.org/10.1007/978-3-540-87442-3_69

2008-01-01

Abstract:Gaussian mixture has been widely used for data modeling and analysis and the EM algorithm is generally employed for its parameter learning. However, the EM algorithm may be trapped into a local maximum of the likelihood and even leads to a wrong result if the number of components is not appropriately set. Recently, the competitive EM (CEM) algorithm for Gaussian mixtures, a new kind of split-and-merge learning algorithm with certain competitive mechanism on estimated components of the EM algorithm, has been constructed to overcome these drawbacks. In this paper, we construct a new CEM algorithm through the Bayesian Ying-Yang (BYY) harmony stop criterion, instead of the previously used MML criterion. It is demonstrated by the simulation experiments that our proposed CEM algorithm outperforms the original one on both model selection and parameter estimation.
Distributed Clustering Using Distributed Mixture of Probabilistic Pca

Hang Yin,Chunhong Zhang,Yang Ji

DOI: https://doi.org/10.1109/fskd.2014.6980859

2014-01-01

Abstract:This paper considers the clustering algorithm based on mixture of probabilistic principle component analyzers in distributed environments. The EM procedure of it is first transformed to a summing variant. Following the classic distributed EM framework for mixture of exponential family distributions and utilizing the summing variant, we propose the distributed EM for mixture of probabilistic principle component analyzers. The proposed algorithm avoids transferring all data from distributed nodes to a central node. Experiment verifies the validity and feasibility of the proposed method. For some datasets, the proposed method can even enhance the log-likelihood as well as the clustering performance.
An Efficient Em Approach To Parameter Learning Of The Mixture Of Gaussian Processes

Yan Yang,Jinwen Ma

DOI: https://doi.org/10.1007/978-3-642-21090-7_20

2011-01-01

Abstract:The mixture of Gaussian processes (MOP) is an important probabilistic model which is often applied to the regression and classification of temporal data. But the existing EM algorithms for its parameter learning encounters a hard difficulty on how to compute the expectations of those assignment variables (as the hidden ones). In this paper, we utilize the leave-one-out cross-validation probability decomposition for the conditional probability and develop an efficient EM algorithm for the MOP model in which the expectations of the assignment variables can be solved directly in the E-step. In the M-step, a conjugate gradient method under a standard Wolfe-Powell line search is implemented to learn the parameters. Furthermore, the proposed EM algorithm can be carried out in a hard cutting way such that each data point is assigned to the GP expert with the highest posterior in the E-step and then the parameters of each GP expert can be learned with these assigned data points in the M-step. Therefore, it has a potential advantage of handling large datasets in comparison with those soft cutting methods. The experimental results demonstrate that our proposed EM algorithm is effective and efficient.
Big Learning Expectation Maximization

Yulai Cong,Sijia Li

2023-12-19

Abstract:Mixture models serve as one fundamental tool with versatile applications. However, their training techniques, like the popular Expectation Maximization (EM) algorithm, are notoriously sensitive to parameter initialization and often suffer from bad local optima that could be arbitrarily worse than the optimal. To address the long-lasting bad-local-optima challenge, we draw inspiration from the recent ground-breaking foundation models and propose to leverage their underlying big learning principle to upgrade the EM. Specifically, we present the Big Learning EM (BigLearn-EM), an EM upgrade that simultaneously performs joint, marginal, and orthogonally transformed marginal matchings between data and model distributions. Through simulated experiments, we empirically show that the BigLearn-EM is capable of delivering the optimal with high probability; comparisons on benchmark clustering datasets further demonstrate its effectiveness and advantages over existing techniques. The code is available at <a class="link-external link-https" href="https://github.com/YulaiCong/Big-Learning-Expectation-Maximization" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Methodology
A Precise Hard-Cut EM Algorithm for Mixtures of Gaussian Processes.

Ziyi Chen,Jinwen Ma,Yatong Zhou

DOI: https://doi.org/10.1007/978-3-319-09339-0_7

2014-01-01

Abstract:The mixture of Gaussian processes (MGP) is a powerful framework for machine learning. However, its parameter learning or estimation is still a very challenging problem. In this paper, a precise hard-cut EM algorithm is proposed for learning the parameters of the MGP without any approximation in the derivation. It is demonstrated by the experimental results that our proposed hard-cut EM algorithm for MGP is feasible and even outperforms two available hard-cut EM algorithms.
Feature Weighted Rival Penalized EM for Gaussian Mixture Clustering: Automatic Feature and Model Selections in a Single Paradigm

Yiu-ming Cheung,Hong Zeng

DOI: https://doi.org/10.1109/iccias.2006.294213

2006-01-01

Abstract:The rival penalized expectation-maximization (RPEM) algorithm has demonstrated its powerful capability to perform the model selection automatically in the context of mixture model. However, the performance may be degraded when irrelevant variables are included. To overcome this drawback, we adopt the concept of feature salience as the feature weight to measure the relevance to the clusters in the subspace, and integrate it into the RPEM algorithm. The proposed algorithm distinguishes the probably redundant features and estimates the number of clusters automatically and simultaneously in a single learning paradigm. Experiments conducted on both synthetic and benchmark real data set have shown the efficacy of the proposed algorithm
The EM Algorithm for Generalized Exponential Mixture Model

Yueyang Teng,Tie Zhang

DOI: https://doi.org/10.1109/cise.2010.5677272

2010-01-01

Abstract:Gaussian mixture is an important distribution to describe mixture data, in which each part mixed must be assumed to be normal. But the normal distribution may not be able to model the data adequately for some cases, so that Gaussian mixture is invalid. The generalized exponential distribution is a flexible density model, which can describe uniform, Gaussian, Laplacian and other sub- and super-Gaussian unimodal densities. In this paper, the generalized exponential mixture model is studied, in which Gaussian mixture and Laplacian mixture are two special cases. Different distributions can also be mixed, such as mixing a Laplacian with a Gaussian. In 1-dimension real space, the EM algorithm is developed to address the solution to the mixture model. The EM algorithm can be conditionally popularized to n-dimension real space. Numerical simulations and image segmentation experiment demonstrate feasibility and effectiveness of the proposed approach.
Probabilistic Two-Dimensional Principal Component Analysis and Its Mixture Model for Face Recognition

Haixian Wang,Sibao Chen,Zilan Hu,Bin Luo

DOI: https://doi.org/10.1007/s00521-007-0155-1

2008-01-01

Neural Computing and Applications

Abstract:Recently, two-dimensional principal component analysis (2DPCA) as a novel eigenvector-based method has proved to be an efficient technique for image feature extraction and representation. In this paper, by supposing a parametric Gaussian distribution over the image space (spanned by the row vectors of 2D image matrices) and a spherical Gaussian noise model for the image, we endow the 2DPCA with a probabilistic framework called probabilistic 2DPCA (P2DPCA), which is robust to noise. Further, by using the probabilistic perspective of P2DPCA, we extend the P2DPCA to a mixture of local P2DPCA models (MP2DPCA). The MP2DPCA offers us a method of being able to model faces in unconstrained (complex) environment. The model parameters could be fitted on the basis of maximum likelihood (ML) estimation via the expectation maximization (EM) algorithm. The experimental recognition results on UMIST, AR face database, and the face recognition (FR) data collected at University of Essex confirm the effectivity of the proposed methods.
Feature Extraction Based on Mixture Probabilistic Kernel Principal Component Analysis

Huibo Zhao,Quan Pan,Yongmei Cheng

DOI: https://doi.org/10.1109/IFITA.2009.11

2009-01-01

Abstract:Feature extraction of training samples and testing samples face the problem of the high non-linear by complexity of the distribution of the samples. In contrast to linear PCA, KPCA is capable of capturing part of the higher-order statistics which are particularly important for encoding image structure. The Probabilistic kernel principal component analysis (PKPCA), defines PPCA probability model by non-linear mapping in the high-dimensional feature space. This paper presents the mix model of the probability of kernel principal component analysis (MPKPCA) method, which adopt a non-linear mapping to make the data from low-dimensional space to the high-dimensional kernel space, in kernel space, using the mixed probability principal component analysis (MPPCA), it combines the advantages of kernel principal component analysis (KPCA) and MPPCA characteristics. Experimental results under complex scenery demonstrate that the proposed algorithm is feasibility and effectiveness.

An Effective Em Algorithm for Pca Mixture Model

Multimode Process Monitoring with PCA Mixture Model

Repulsive Mixture Models of Exponential Family PCA for Clustering

Competitive EM Algorithm for Finite Mixture Models

An Mcmc Based Em Algorithm For Mixtures Of Gaussian Processes

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Robust modeling of mixture probabilistic principal component analysis and process monitoring application

HeMPPCAT: Mixtures of Probabilistic Principal Component Analysers for Data with Heteroscedastic Noise

An Effective EM Algorithm for Mixtures of Gaussian Processes Via the MCMC Sampling and Approximation.

Automatic T-Mixture Model Selection Via Rival Penalized EM

An EM Algorithm for Robust Bayesian PCA with Student’s T-Distribution

On Semi-Supervised Learning Genetic-Based And Deterministic Annealing Em Algorithm For Dirichlet Mixture Models

The Competitive EM Algorithm for Gaussian Mixtures with BYY Harmony Criterion

Distributed Clustering Using Distributed Mixture of Probabilistic Pca

An Efficient Em Approach To Parameter Learning Of The Mixture Of Gaussian Processes

Big Learning Expectation Maximization

A Precise Hard-Cut EM Algorithm for Mixtures of Gaussian Processes.

Feature Weighted Rival Penalized EM for Gaussian Mixture Clustering: Automatic Feature and Model Selections in a Single Paradigm

The EM Algorithm for Generalized Exponential Mixture Model

Probabilistic Two-Dimensional Principal Component Analysis and Its Mixture Model for Face Recognition

Feature Extraction Based on Mixture Probabilistic Kernel Principal Component Analysis