Abstract:Mixed multinomial logits are discrete mixtures introduced several decades ago to model the probability of choosing an attribute from $p$ possible candidates, in heterogeneous populations. The model has recently attracted attention in the AI literature, under the name softmax mixtures, where it is routinely used in the final layer of a neural network to map a large number $p$ of vectors in $\mathbb{R}^L$ to a probability vector. Despite its wide applicability and empirical success, statistically optimal estimators of the mixture parameters, obtained via algorithms whose running time scales polynomially in $L$, are not known. This paper provides a solution to this problem for contemporary applications, such as large language models, in which the mixture has a large number $p$ of support points, and the size $N$ of the sample observed from the mixture is also large. Our proposed estimator combines two classical estimators, obtained respectively via a method of moments (MoM) and the expectation-minimization (EM) algorithm. Although both estimator types have been studied, from a theoretical perspective, for Gaussian mixtures, no similar results exist for softmax mixtures for either procedure. We develop a new MoM parameter estimator based on latent moment estimation that is tailored to our model, and provide the first theoretical analysis for a MoM-based procedure in softmax mixtures. Although consistent, MoM for softmax mixtures can exhibit poor numerical performance, as observed other mixture models. Nevertheless, as MoM is provably in a neighborhood of the target, it can be used as warm start for any iterative algorithm. We study in detail the EM algorithm, and provide its first theoretical analysis for softmax mixtures. Our final proposal for parameter estimation is the EM algorithm with a MoM warm start.

TLS-EM Algorithm of Mixture Density Models for Exponential Families

Regularized TLS-EM for estimating missing data

Model-based clustering and classification using mixtures of multivariate skewed power exponential distributions

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

Asymptotic Convergence Properties of the EM Algorithm with Respect to the Overlap in the Mixture

Competitive EM Algorithm for Finite Mixture Models

On Weight Design of Maximum Weighted Likelihood and an Extended EM Algorithm

Big Learning Expectation Maximization

On-line Evolutionary Exponential Family Mixture

A Novel Split and Merge EM Algorithm for Gaussian Mixture Model

An Effective EM Algorithm for Mixtures of Gaussian Processes Via the MCMC Sampling and Approximation.

A Robust and Flexible EM Algorithm for Mixtures of Elliptical Distributions with Missing Data

An Mcmc Based Em Algorithm For Mixtures Of Gaussian Processes

Statistical Machine Learning EM algorithm Lecture Notes 13 : EM algorithm

Learning large softmax mixtures with warm start EM

ML-ESTIMATION OF THE T-DISTRIBUTION USING EM AND ITS EXTENSIONS, ECM AND ECME

Additive mixed models with approximate Dirichlet process mixtures: the EM approach

Network EM Algorithm for Gaussian Mixture Model in Decentralized Federated Learning

The Competitive EM Algorithm for Gaussian Mixtures with BYY Harmony Criterion

Benefits of over-parameterization with EM

Statistical Convergence of the EM Algorithm on Gaussian Mixture Models