Abstract:We employ a parameter-free distribution estimation framework where estimators are random distributions and utilize the Kullback-Leibler (KL) divergence as a loss function. Wu and Vos [J. Statist. Plann. Inference 142 (2012) 1525-1536] show that when an estimator obtained from an i.i.d. sample is viewed as a random distribution, the KL risk of the estimator decomposes in a fashion parallel to the mean squared error decomposition when the estimator is a real-valued random variable. In this paper, we explore how conditional versions of distribution expectation ($E^{\dagger}$) can be defined so that a distribution version of the Rao-Blackwell theorem holds. We define distributional expectation and variance ($V^{\dagger}$) that also provide a decomposition of KL risk in exponential and mixture families. For exponential families, we show that the maximum likelihood estimator (viewed as a random distribution) is distribution unbiased and is the unique uniformly minimum distribution variance unbiased (UMV$^{\dagger}$U) estimator. Furthermore, we show that the MLE is robust against model specification in that if the true distribution does not belong to the exponential family, the MLE is UMV$^{\dagger}$U for the KL projection of the true distribution onto the exponential families provided these two distribution have the same expectation for the canonical statistic. To allow for estimators taking values outside of the exponential family, we include results for KL projection and define an extended projection to accommodate the non-existence of the MLE for families having discrete sample space. Illustrative examples are provided.

Maximum Likelihood Estimates and Likelihood Ratio Criteria for Location and Scale Parameters of the Multivariatel 1-Norm Symmetric Distributions

Maximum Likelihood Estimators and Likelihood Ratio Criteria for Multivariate Elliptically Contoured Distributions

On mean and/or variance mixtures of normal distributions

Statistical Inference for the Location and Scale Parameters of the Skew Normal Distribution

The Maximum Likelihood Estimation with the Order Statistics

A Constrained Maximum Likelihood Estimation for Skew Normal Mixtures

Maximum Likelihood Character of Distributions

Maximum-Likelihood-Estimation of Parameters under A Spatial Sampling Scheme

Extention of the Invariance of Maximum Likelihood Estimation

Existence of Maximum Likelihood Estimation for Three-Parameter Log-Normal Distribution

A Characterization of Multivariate L1-Norm Symmetric Distributions

Properties and maximum likelihood estimation of the gamma-normal and related probability distributions

Normalized Maximum Likelihood with Luckiness for Multivariate Normal Distributions

Nonparametric maximum likelihood estimation under a likelihood ratio order

Maximum likelihood estimators uniformly minimize distribution variance among distribution unbiased estimators in exponential families

Rational Maximum Likelihood Estimators of Kronecker Covariance Matrices

Moderate Deviation Principles for Classical Likelihood Ratio Tests of High-Dimensional Normal Distributions.

Univariate likelihood projections and characterizations of the multivariate normal distribution

A Note on Asymptotics of Classical Likelihood Ratio Tests for High-Dimensional Normal Distributions

Estimation and Influence Diagnostics for the Multivariate Linear Regression Models with Skew Scale Mixtures of Normal Distributions

Joint Test for Homogeneity of High-Dimensional Means and Covariance Matrices Using Maximum-Type Statistics