Abstract:In density estimation task, Maximum Entropy Maxent model can effectively use reliable prior information via nonparametric constraints, that is, linear constraints without empirical parameters. However, reliable prior information is often insufficient, and parametric constraints becomes necessary but poses considerable implementation complexity. Improper setting of parametric constraints can result in overfitting or underfitting. To alleviate this problem, a generalization of Maxent, under Tsallis entropy framework, is proposed. The proposed method introduces a convex quadratic constraint for the correction of expected quadratic Tsallis Entropy Bias TEB. Specifically, we demonstrate that the expected quadratic Tsallis entropy of sampling distributions is smaller than that of the underlying real distribution with regard to frequentist, Bayesian prior, and Bayesian posterior framework, respectively. This expected entropy reduction is exactly the expected TEB, which can be expressed by the closed-form formula and acts as a consistent and unbiased correction with an appropriate convergence rate. TEB indicates that the entropy of a specific sampling distribution should be increased accordingly. This entails a quantitative reinterpretation of the Maxent principle. By compensating TEB and meanwhile forcing the resulting distribution to be close to the sampling distribution, our generalized quadratic Tsallis Entropy Bias Compensation TEBC Maxent can be expected to alleviate the overfitting and underfitting. We also present a connection between TEB and Lidstone estimator. As a result, TEB-Lidstone estimator is developed by analytically identifying the rate of probability correction in Lidstone. Extensive empirical evaluation shows promising performance of both TEBC Maxent and TEB-Lidstone in comparison with various state-of-the-art density estimation methods.

QUADRATIC TSALLIS ENTROPY BIAS AND GENERALIZED MAXIMUM ENTROPY MODELS

Minimum Total Error Entropy Method for Parameter Estimation

A Maximum Entropy Copula Model for Mixed Data: Representation, Estimation, and Applications

MESSY Estimation: Maximum-Entropy based Stochastic and Symbolic densitY Estimation

Robust Discriminant Analysis Based on Nonparametric Maximum Entropy

A New Maximum Entropy Method for Estimation of Multimodal Probability Density Function

The Principle of Uncertain Maximum Entropy

Improving Decision Trees by Tsallis Entropy Information Metric Method.

Tsallis eXtropy

Nonextensive Belief Entropy

Entropy Estimation via Uniformization

Principal Component Analysis Based on Non-Parametric Maximum Entropy

Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts

Maximum Probability and Relative Entropy Maximization. Bayesian Maximum Probability and Empirical Likelihood

Maximum Entropy Estimator for Hidden Markov Models: Reduction to Dimension 2

Approximate maximum entropy principles via Goemans-Williamson with applications to provable variational methods

Modeling Images and Textures by Minimax Entropy.

Improved Image Thresholding Based on 2-D Tsallis Entropy

Maximum Entropy Principle, Equal Probability a Priori and Gibbs Paradox

Generalized Maximum Entropy Differential Dynamic Programming

Entropy Maximization with Linear Constraints: The Uniqueness of the Shannon Entropy