Abstract:We develop a class of interacting particle systems for implementing a maximum marginal likelihood estimation (MMLE) procedure to estimate the parameters of a latent variable model. We achieve this by formulating a continuous-time interacting particle system which can be seen as a Langevin diffusion over an extended state space of parameters and latent variables. In particular, we prove that the parameter marginal of the stationary measure of this diffusion has the form of a Gibbs measure where number of particles acts as the inverse temperature parameter in classical settings for global optimisation. Using a particular rescaling, we then prove geometric ergodicity of this system and bound the discretisation error in a manner that is uniform in time and does not increase with the number of particles. The discretisation results in an algorithm, termed Interacting Particle Langevin Algorithm (IPLA) which can be used for MMLE. We further prove nonasymptotic bounds for the optimisation error of our estimator in terms of key parameters of the problem, and also extend this result to the case of stochastic gradients covering practical scenarios. We provide numerical experiments to illustrate the empirical behaviour of our algorithm in the context of logistic regression with verifiable assumptions. Our setting provides a straightforward way to implement a diffusion-based optimisation routine compared to more classical approaches such as the Expectation Maximisation (EM) algorithm, and allows for especially explicit nonasymptotic bounds.

Expectation-Maximization-Maximization: A Feasible MLE Algorithm for the Three-Parameter Logistic Model Based on a Mixture Modeling Reformulation

Maximum Likelihood Estimators in a Two Step Model for PLS

The Improved EMS Algorithm for Latent Variable Selection in M3PL Model

Accelerating L1-penalized expectation maximization algorithm for latent variable selection in multidimensional two-parameter logistic models

On Efficient and Scalable Computation of the Nonparametric Maximum Likelihood Estimator in Mixture Models

Big Learning Expectation Maximization

Bayesian or Non-Bayesian: A Comparison Study of Item Parameter Estimation in the Three-Parameter Logistic Model

Maximum-Likelihood-Estimation Via the Ecm Algorithm - A General Framework

Random pairing MLE for estimation of item parameters in Rasch model

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Joint Maximum Likelihood Estimation for High-Dimensional Exploratory Item Factor Analysis

Modified generalized method of moments for a robust estimation of polytomous logistic model

Joint Maximum Likelihood Estimation for High-dimensional Exploratory Item Response Analysis

Interacting Particle Langevin Algorithm for Maximum Marginal Likelihood Estimation

Approaches in Using Expectation Maximization Algorithm for Maximum Likelihood Estimation of the Parameters of a Constrained State Space Model with an External Input Series

Learning Mixed Multinomial Logits with Provable Guarantees

An Mpec Estimator for Misclassification Models

An Efficient Em Approach To Parameter Learning Of The Mixture Of Gaussian Processes

Fast ML Estimation for the Mixture of Factor Analyzers Via an ECM Algorithm

The Rank-2PL IRT Models for Forced-Choice Questionnaires: Maximum Marginal Likelihood Estimation with an EM Algorithm

Semi-supervised Logistic Learning Based on Exponential Tilt Mixture Models