Abstract:Mixture of experts (ME) is a modular neural network architecture for supervised classification. The double-loop expectation-maximization (EM) algorithm has been developed for learning the parameters of the ME architecture, and the iteratively reweighted least squares (IRLS) algorithm and the Newton-Raphson algorithm are two popular schemes for learning the parameters in the inner loop or gating network. In this letter, we investigate asymptotic convergence properties of the EM algorithm for ME using either the IRLS or Newton-Raphson approach. With the help of an overlap measure for the ME model, we obtain an upper bound of the asymptotic convergence rate of the EM algorithm in each case. Moreover, we find that for the Newton approach as a specific Newton-Raphson approach to learning the parameters in the inner loop, the upper bound of asymptotic convergence rate of the EM algorithm locally around the true solution Θ* is [Formula: see text], where ϵ>0 is an arbitrarily small number, o(x) means that it is a higher-order infinitesimal as x → 0, and e(Θ*) is a measure of the average overlap of the ME model. That is, as the average overlap of the true ME model with large sample tends to zero, the EM algorithm with the Newton approach to learning the parameters in the inner loop tends to be asymptotically superlinear. Finally, we substantiate our theoretical results by simulation experiments.

Asymptotic Convergence Properties of Entropy Regularized Likelihood Learning on Finite Mixtures with Automatic Model Selection

Unsupervised learning of finite mixtures using entropy regularization and its application to image segmentation

Letters: An iterative algorithm for entropy regularized likelihood learning on Gaussian mixture with automatic model selection

A gradient entropy regularized likelihood learning algorithm on gaussian mixture with automatic model selection

Entropy Regularized Likelihood Learning on Gaussian Mixture: Two Gradient Implementations for Automatic Model Selection

A generalized competitive learning algorithm on gaussian mixture with automatic model selection

A Semi-Supervised Learning Algorithm on Gaussian Mixture with Automatic Model Selection

Generalized Competitive Learning of Gaussian Mixture Models

Learning Mixture Models with the Regularized Latent Maximum Entropy Principle

An Iterative Entropy Regularized Likelihood Learning Algorithm for Cluster Analysis with the Number of Clusters Automatically Detected

Model Selection for Exponential Power Mixture Regression Models

Mixture Bayesian Regularization of PCR Model and Soft Sensing Application

Dynamically regularized maximum likelihood learning of Gaussian mixtures

Harmonious Competition Learning for Gaussian Mixtures

Entropy Regularization, Automatic Model Selection, And Unsupervised Image Segmentation

On Semi-Supervised Learning Genetic-Based And Deterministic Annealing Em Algorithm For Dirichlet Mixture Models

Asymptotic Properties of Nonparametric M-Estimation for Mixing Functional Data

Asymptotic Convergence Properties of the Em Algorithm for Mixture of Experts

Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models

Entropy Penalized Automated Model Selection On Gaussian Mixture

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models