Abstract:We develop a class of interacting particle systems for implementing a maximum marginal likelihood estimation (MMLE) procedure to estimate the parameters of a latent variable model. We achieve this by formulating a continuous-time interacting particle system which can be seen as a Langevin diffusion over an extended state space of parameters and latent variables. In particular, we prove that the parameter marginal of the stationary measure of this diffusion has the form of a Gibbs measure where number of particles acts as the inverse temperature parameter in classical settings for global optimisation. Using a particular rescaling, we then prove geometric ergodicity of this system and bound the discretisation error in a manner that is uniform in time and does not increase with the number of particles. The discretisation results in an algorithm, termed Interacting Particle Langevin Algorithm (IPLA) which can be used for MMLE. We further prove nonasymptotic bounds for the optimisation error of our estimator in terms of key parameters of the problem, and also extend this result to the case of stochastic gradients covering practical scenarios. We provide numerical experiments to illustrate the empirical behaviour of our algorithm in the context of logistic regression with verifiable assumptions. Our setting provides a straightforward way to implement a diffusion-based optimisation routine compared to more classical approaches such as the Expectation Maximisation (EM) algorithm, and allows for especially explicit nonasymptotic bounds.

On the maximum likelihood estimation in general log-linear models

Maximum Likelihood Estimators in a Two Step Model for PLS

Essential formulae for restricted maximum likelihood and its derivatives associated with the linear mixed models

On the correspondence of deviances and maximum likelihood and interval estimates from log-linear to logistic regression modelling

Maximum Approximated Likelihood Estimation

Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model

Finite-sample performance of the maximum likelihood estimator in logistic regression

Maximum Ideal Likelihood Estimator: An New Estimation and Inference Framework for Latent Variable Models

Strong Consistency of Maximum Quasi-Likelihood Estimators in Generalized Linear Models with Fixed and Adaptive Designs

Empirical Likelihood for Generalized Linear Models with Longitudinal Data

Computing log-likelihood and its derivatives for restricted maximum likelihood methods

Restricted maximum likelihood estimation in generalized linear mixed models

Decorrelated Empirical Likelihood for Generalized Linear Models with High-Dimensional Longitudinal Data

A modern maximum-likelihood theory for high-dimensional logistic regression

Robustified Maximum Likelihood Estimation in Generalized Partial Linear Mixed Model for Longitudinal Data

Exact Solutions in Log-Concave Maximum Likelihood Estimation

Weak Consistency of Quasi-Maximum Likelihood Estimate in Generalized Linear Models

A proof of consistency and model-selection optimality on the empirical Bayes method

Interacting Particle Langevin Algorithm for Maximum Marginal Likelihood Estimation

Covariance Model with General Linear Structure and Divergent Parameters

On variational inference and maximum likelihood estimation with the λ-exponential family